Skip to content

I
Ip2region

int / Ip2region
上一次同步 11 个月

通知 19
Star 0
Fork 0
I
Ip2region

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

源分支
选择Git版本
...
目标分支
选择Git版本
 
Commits (197)
展开全部 隐藏空白更改
内联 并排
Showing with 698829 addition and 285 deletion
.gitignore
浏览文件 @ 65edd65b
...@@ -6,12 +6,19 @@ ...@@ -6,12 +6,19 @@
*.log *.log
*.la *.la
*.so *.so
*.xdb
*.iml
META-INF/ META-INF/
.DS_Store
# Binary Files # # Binary Files #
*.jar *.jar
!dbMaker-*.jar !dbMaker-*.jar
# ignore all xdb except the one in ./data/
*.xdb
!/data/*.xdb
# Package Files # # Package Files #
.settings/ .settings/
.classpath .classpath
...@@ -22,12 +29,27 @@ META-INF/ ...@@ -22,12 +29,27 @@ META-INF/
.idea .idea
# binding # binding
/v1.0//binding/java/classes/
/v1.0//binding/java/doc/
/v1.0//binding/java/target/
/v1.0//binding/java/*.jar
/binding/java/classes/ /binding/java/classes/
/binding/java/doc/ /binding/java/doc/
/binding/java/target/ /binding/java/target/
/binding/java/*.jar /binding/java/*.jar
/binding/c/testSearcher # clang
/binding/c/xdb_searcher
/binding/c/util_test
/binding/c/cmake-build-debug
# lua/luc_c
/binding/lua_c/cmake-build-debug
# golang
/binding/golang/searcher
/binding/golang/xdb_searcher
/binding/golang/golang
# rust # rust
Cargo.lock Cargo.lock
...@@ -36,11 +58,26 @@ target ...@@ -36,11 +58,26 @@ target
# VS ignore cases # VS ignore cases
/**/*.sln /**/*.sln
/v1.0/binding/c#/**/.vs/
/v1.0/binding/c#/**/packages
/v1.0/binding/c#/**/bin
/v1.0/binding/c#/**/obj
/binding/c#/**/.vs/ /binding/c#/**/.vs/
/binding/c#/**/packages /binding/c#/**/packages
/binding/c#/**/bin /binding/c#/**/bin
/binding/c#/**/obj /binding/c#/**/obj
# Nodejs # Nodejs
/v1.0/binding/nodejs/tests/unitTests/__snapshots__
/v1.0/binding/nodejs/coverage
/binding/nodejs/tests/unitTests/__snapshots__ /binding/nodejs/tests/unitTests/__snapshots__
/binding/nodejs/coverage /binding/nodejs/coverage
/binding/nodejs/node_modules
/binding/nodejs/.nyc_output
/binging/nodejs/package-lock.json
# maker
## golang
/maker/golang/dbmaker
/maker/golang/xdb_maker
/maker/golang/golang
ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# Ip2region 是什么
ip2region v2.0 - 是一个离线IP地址定位库和IP定位数据管理框架，10微秒级别的查询效率，提供了众多主流编程语言的 `xdb` 数据生成和查询客户端实现。v1.0 旧版本: [v1.0版本入口](v1.0)
# Ip2region 特性
### 1、标准化的数据格式
每个 ip 数据段的 region 信息都固定了格式：`国家|区域|省份|城市|ISP`，只有中国的数据绝大部分精确到了城市，其他国家部分数据只能定位到国家，后前的选项全部是0。
### 2、数据去重和压缩
`xdb` 格式生成程序会自动去重和压缩部分数据，默认的全部 IP 数据，生成的 ip2region.xdb 数据库是 11MiB，随着数据的详细度增加数据库的大小也慢慢增大。
### 3、极速查询响应
即使是完全基于 `xdb` 文件的查询，单次查询响应时间在十微秒级别，可通过如下两种方式开启内存加速查询：
1. `vIndex` 索引缓存 ：使用固定的 `512KiB` 的内存空间缓存 vector index 数据，减少一次 IO 磁盘操作，保持平均查询效率稳定在10-20微秒之间。
2. `xdb` 整个文件缓存：将整个 `xdb` 文件全部加载到内存，内存占用等同于 `xdb` 文件大小，无磁盘 IO 操作，保持微秒级别的查询效率。
### 4、IP 数据管理框架
v2.0 格式的 `xdb` 支持亿级别的 IP 数据段行数，region 信息也可以完全自定义，例如：你可以在 region 中追加特定业务需求的数据，例如：GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。
# `xdb` 数据查询
API 介绍，使用文档和测试程序请参考对应 `searcher` 查询客户端下的 ReadMe 介绍，全部查询 binding 实现情况如下：
| Ok? | 状态 | 编程语言 | 描述 | 贡献者 |
|:-------------------|:-----| :--- |:----------------------|:--------------------|
| :white_check_mark: | 已完成 | [golang](binding/golang) | golang xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [php](binding/php) | php xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [java](binding/java) | java xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [lua](binding/lua) | 纯 lua xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [c](binding/c) | ANSC c xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [lua_c](binding/lua_c) | lua c 扩展 xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
|     | 待开始 | [rust](binding/rust) | rust xdb 查询客户端实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [python](binding/python) | python xdb 查询客户端实现 | [厉害的花花](https://github.com/luckydog6132) |
| :white_check_mark: | 已完成 | [nodejs](binding/nodejs) | nodejs xdb 查询客户端实现 | [Wu Jian Ping](https://github.com/wujjpp) |
| :white_check_mark: | 已完成 | [csharp](binding/csharp) | csharp xdb 查询客户端实现 | [Soar360](https://gitee.com/Soar360) |
|     | 待开始 | [php_ext](binding/php7_ext) | php c 扩展 xdb 查询客户端实现 | 待确定 |
|     | 待开始 | [nginx](binding/nginx) | nginx 扩展 xdb 查询客户端实现 | 待确定 |
# `xdb` 数据生成
API 介绍，使用文档和测试程序请参考对应 `maker` 生成程序下的 ReadMe 介绍，全部生成 maker 实现情况如下：
| Ok? | 状态 | 编程语言 | 描述 | 贡献者 |
|:-------------------|:-----| :--- | :--- | :--- |
| :white_check_mark: | 已完成 | [golang](maker/golang) | golang xdb 生成程序实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [java](maker/java) | java xdb 生成程序实现 | [Lion](https://github.com/lionsoul2014) |
|     | 待开始 | [c](maker/c) | ANSC c xdb 生成程序实现 | [Lion](https://github.com/lionsoul2014) |
| :white_check_mark: | 已完成 | [python](maker/python) | python xdb 生成程序实现 | [leolin49](https://github.com/leolin49) |
# 并发查询必读
全部查询客户端的 search 接口都 <b>不是</b> 并发安全的实现，不同进程/线程/协程需要通过创建不同的查询对象来安全使用，并发量很大的情况下，基于文件查询的方式可能会导致打开文件数过多的错误，请修改内核的最大允许打开文件数(fs.file-max=一个更高的值)，或者将整个xdb加载到内存进行安全并发使用。
# 相关备注
### 1、使用声明
ip2region 重点在于<b>研究 IP 定位数据的存储设计和各种语言的查询实现</b>，并没有原始 IP 数据的支撑，本项目不保证及时的数据更新，没有也不会有商用版本，你可以使用自定义的数据导入 ip2region 进行管理。
### 2、技术交流
ip2region 微信交流群，请先加微信：lionsoul2014 (请备注 ip2region)
### 3、数据更新
基于检测算法的数据更新方式视频分享：[数据更新实现视频分享 - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)，[数据更新实现视频分享 - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
### 4、数据结构
1. xdb 数据结构分析：[“ip2region xdb 数据结构和查询过程的详解“](https://mp.weixin.qq.com/s?__biz=MzU4MDc2MzQ5OA==&mid=2247483696&idx=1&sn=6e9e138e86cf18245656c54ff4be3129&chksm=fd50ab35ca2722239ae7c0bb08efa44f499110c810227cbad3a16f36ebc1c2afc58eb464a57c#rd)
2. xdb 查询过程分析：[“ip2region xdb 数据结构和查询过程的详解”](https://mp.weixin.qq.com/s?__biz=MzU4MDc2MzQ5OA==&mid=2247483696&idx=1&sn=6e9e138e86cf18245656c54ff4be3129&chksm=fd50ab35ca2722239ae7c0bb08efa44f499110c810227cbad3a16f36ebc1c2afc58eb464a57c#rd)
binding/c/Makefile
浏览文件 @ 65edd65b
A = @ all: xdb_searcher util_test
CFLAGS= -g
CC= gcc
LIBS= -I./
LDFLAGS=
RM = rm -f
MV = mv
testSearcher: ip2region.o xdb_searcher: xdb_searcher.h xdb_searcher.c main.c
$(A) $(CC) $(CFLAGS) $(LDFLAGS) $(LIBS) -o testSearcher testSearcher.c ip2region.c gcc -g -O2 -I./ xdb_searcher.c main.c -o xdb_searcher
$(A) $(RM) ip2region.o testSearcher.o
db:
$(A) cd ../../;\
export LANG=en_US.UTF-8;\
java -jar dbMaker-1.2.2.jar -src ./data/ip.merge.txt -region ./data/global_region.csv -dst .;\
$(MV) ./ip2region.db ./binding/c/;\
cd ./binding/c
all: db testSearcher
util_test: xdb_searcher.h xdb_searcher.c util_test.c
gcc -g -O2 -I./ xdb_searcher.c util_test.c -o util_test
clean: clean:
$(A) $(RM) testSearcher find ./ -name \*.o | xargs rm -f
find ./ -name util_test | xargs rm -f
.PHONY: all clean testSearcher db find ./ -name xdb_searcher | xargs rm -f
.PHONY: all clean xdb_searcher util_test
\ No newline at end of file
binding/c/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# ip2region xdb c 查询客户端实现
# 使用方式
### 完全基于文件的查询
```c
#include <stdio.h>
#include "xdb_searcher.h"
int main(int argc, char *argv[]) {
char *db_path = "ip2region.xdb file path";
xdb_searcher_t searcher;
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
long s_time;
// 1、从 db_path 初始化 xdb 查询对象
int err = xdb_new_with_file_only(&searcher, db_path);
if (err != 0) {
printf("failed to create xdb searcher from `%s` with errno=%d\n", db_path, err);
return 1;
}
// 2、调用 search API 查询
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip, err);
} else {
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
}
// 备注：并发使用，没一个线程需要单独定义并且初始化一个 searcher 查询对象。
// 3、关闭 xdb 查询器
xdb_close(&searcher);
return 0;
}
```
### 缓存 `VectorIndex` 索引
我们可以提前从 xdb 文件中加载出来 VectorIndex 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
```c
#include <stdio.h>
#include "xdb_searcher.h"
int main(int argc, char *argv[]) {
char *db_path = "ip2region.xdb file path";
xdb_vector_index_t *v_index;
xdb_searcher_t searcher;
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
long s_time;
// 1、从 db_path 加载 VectorIndex 索引。
// 得到 v_index 做成全局缓存，便于后续反复使用。
// 注意：v_index 不需要每次都加载，建议在服务启动的时候加载一次，然后做成全局资源。
v_index = xdb_load_vector_index_from_file(db_path);
if (v_index == NULL) {
printf("failed to load vector index from `%s`\n", db_path);
return 1;
}
// 2、使用全局的 VectorIndex 变量创建带 VectorIndex 缓存的 xdb 查询对象
int err = xdb_new_with_vector_index(&searcher, db_path, v_index);
if (err != 0) {
printf("failed to create vector index cached searcher with errcode=%d\n", err);
return 2;
}
// 3、调用 search API 查询
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip, err);
} else {
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
}
// 备注：并发使用，没一个线程需要单独定义并且初始化一个 searcher 查询对象。
// 4、关闭 xdb 查询器，如果是要关闭服务，也需要释放 v_index 的内存。
xdb_close(&searcher);
xdb_close_vector_index(v_index);
return 0;
}
```
### 缓存整个 `xdb` 数据
我们也可以预先加载整个 ip2region.xdb 的数据到内存，然后基于这个数据创建查询对象来实现完全基于文件的查询，类似之前的 memory search。
```c
#include <stdio.h>
#include "xdb_searcher.h"
int main(int argc, char *argv[]) {
char *db_path = "ip2region.xdb file path";
xdb_content_t *c_buffer;
xdb_searcher_t searcher;
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
long s_time;
// 1、从 db_path 加载整个 xdb 的数据。
c_buffer = xdb_load_content_from_file(db_path);
if (v_index == NULL) {
printf("failed to load xdb content from `%s`\n", db_path);
return 1;
}
// 2、使用全局的 c_buffer 变量创建一个完全基于内存的 xdb 查询对象
err = xdb_new_with_buffer(&searcher, c_buffer);
if (err != 0) {
printf("failed to create content cached searcher with errcode=%d\n", err);
return 2;
}
// 3、调用 search API 查询
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip, err);
} else {
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
}
// 备注：并发使用，使用这种方式创建的 xdb 查询对象可以安全用于并发。
// 建议在服务启动的时候创建好，然后一直安全并发使用，直到服务关闭。
// 4、关闭 xdb 查询器，关闭服务的时候需要释放 c_buffer 的内存。
xdb_close(&searcher);
xdb_close_content(c_buffer);
return 0;
}
```
# 测试程序编译
通过如下方式编译得到 xdb_searcher 可执行程序：
```bash
# cd 到 c binding 根目录
make
```
# 查询测试
通过 `xdb_searcher search` 命令来测试对 ip2region.xdb 的查询：
```bash
➜ c git:(c_binding) ✗ ./xdb_searcher search
./xdb_searcher search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 进行查询测试：
```bash
➜ c git:(c_binding) ✗ ./xdb_searcher search --db=../../data/ip2region.xdb --cache-policy=vectorIndex
ip2region xdb searcher test program, cache_policy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, io_count: 7, took: 13 μs}
ip2region>>
```
输入 ip 即可进行查询，输入 quit 即可退出测试程序。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同的缓存实现的效率。
# bench 测试
通过 `xdb_searcher bench` 命令来进行 bench 测试，一方面确保查询程序和 `xdb` 文件没有错误，另一方面可以通过大量的查询得到评价的查询性能：
```bash
➜ c git:(c_binding) ✗ ./xdb_searcher bench
./xdb_searcher bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 来进行 bench 测试：
```bash
➜ c git:(c_binding) ✗ ./xdb_searcher bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt --cache-policy=vectorIndex
Bench finished, {cache_policy: vectorIndex, total: 3417955, took: 4.233s, cost: 1 μs/op}
```
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。 @Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件相同的源文件。
binding/c/main.c 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/28
#include "stdio.h"
#include "xdb_searcher.h"
struct searcher_test_entry {
xdb_searcher_t searcher;
xdb_vector_index_t *v_index;
xdb_content_t *c_buffer;
};
typedef struct searcher_test_entry searcher_test_t;
int init_searcher_test(searcher_test_t *test, char *db_path, char *cache_policy) {
int err;
test->v_index = NULL;
test->c_buffer = NULL;
if (strcmp(cache_policy, "file") == 0) {
err = xdb_new_with_file_only(&test->searcher, db_path);
if (err != 0) {
printf("failed to create searcher with errcode=%d\n", err);
return 1;
}
} else if (strcmp(cache_policy, "vectorIndex") == 0) {
test->v_index = xdb_load_vector_index_from_file(db_path);
if (test->v_index == NULL) {
printf("failed to load vector index from `%s`\n", db_path);
return 2;
}
err = xdb_new_with_vector_index(&test->searcher, db_path, test->v_index);
if (err != 0) {
printf("failed to create vector index cached searcher with errcode=%d\n", err);
return 3;
}
} else if (strcmp(cache_policy, "content") == 0) {
test->c_buffer = xdb_load_content_from_file(db_path);
if (test->c_buffer == NULL) {
printf("failed to load xdb content from `%s`\n", db_path);
return 4;
}
err = xdb_new_with_buffer(&test->searcher, test->c_buffer);
if (err != 0) {
printf("failed to create content cached searcher with errcode=%d\n", err);
return 5;
}
} else {
printf("invalid cache policy `%s`, options: file/vectorIndex/content\n", cache_policy);
return 6;
}
return 0;
}
void destroy_searcher_test(searcher_test_t *test) {
xdb_close(&test->searcher);
// check and free the vector index
if (test->v_index != NULL) {
xdb_close_vector_index(test->v_index);
test->v_index = NULL;
}
// check and free the content buffer
if (test->c_buffer != NULL) {
xdb_close_content(test->c_buffer);
test->c_buffer = NULL;
}
}
//read a line from a command line.
static char *get_line(FILE *fp, char *__dst) {
register int c;
register char *cs;
cs = __dst;
while ( ( c = getc( fp ) ) != EOF ) {
if ( c == '\n' ) break;
*cs++ = c;
}
*cs = '\0';
return ( c == EOF && cs == __dst ) ? NULL : __dst;
}
void print_help(char *argv[]) {
printf("ip2region xdb searcher\n");
printf("%s [command] [command options]\n", argv[0]);
printf("Command: \n");
printf(" search search input test\n");
printf(" bench search bench test\n");
}
void test_search(int argc, char *argv[]) {
int i, n, err;
// for args parse
char *r, key[33] = {'\0'}, val[256] = {'\0'};
char db_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
// for search
long s_time, c_time;
unsigned int ip;
char line[512] = {'\0'}, region[512] = {'\0'};
searcher_test_t test;
for (i = 2; i < argc; i++) {
r = argv[i];
if (strlen(r) < 5) {
continue;
}
if (r[0] != '-' || r[1] != '-') {
continue;
}
if (strchr(r, '=') == NULL) {
printf("missing = for args pair '%s'\n", r);
return;
}
n = sscanf(r+2, "%32[^=]=%255[^\n]", key, val);
if (n != 2) {
printf("invalid option flag `%s`\n", r);
return;
}
// printf("key=%s, val=%s\n", key, val);
if (strcmp(key, "db") == 0) {
snprintf(db_file, sizeof(db_file), "%s", val);
} else if (strcmp(key, "cache-policy") == 0) {
memcpy(cache_policy, val, sizeof(cache_policy) - 1);
// snprintf(cache_policy, sizeof(cache_policy), "%s", val);
} else {
printf("undefined option `%s`\n", r);
return;
}
}
if (strlen(db_file) < 1) {
printf("%s search [command options]\n", argv[0]);
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
// printf("db_file=%s, cache_policy=%s\n", db_file, cache_policy);
err = init_searcher_test(&test, db_file, cache_policy);
if (err != 0) {
// init program will print the error reasons;
return;
}
printf("ip2region xdb searcher test program, "
"cache_policy: %s\ntype 'quit' to exit\n", cache_policy);
while ( 1 ) {
printf("ip2region>> ");
get_line(stdin, line);
if ( strlen(line) < 2 ) {
continue;
}
if (strcasecmp( line, "quit") == 0 ) {
break;
}
if (xdb_check_ip(line, &ip) != 0) {
printf("invalid ip address `%s`\n", line);
continue;
}
s_time = xdb_now();
err = xdb_search(&test.searcher, ip, region, sizeof(region));
if (err != 0) {
printf("{err: %d, io_count: %d}\n", err, xdb_get_io_count(&test.searcher));
} else {
c_time = xdb_now() - s_time;
printf("{region: %s, io_count: %d, took: %ld μs}\n", region, xdb_get_io_count(&test.searcher), c_time);
}
}
destroy_searcher_test(&test);
printf("searcher test program exited, thanks for trying\n");
}
void test_bench(int argc, char *argv[]) {
int i, n, err;
char *r, key[33] = {'\0'}, val[256] = {'\0'};
char db_file[256] = {'\0'}, src_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
FILE *handle;
char line[1024] = {'\0'}, sip_str[16] = {'\0'}, eip_str[16] = {'\0'};
char src_region[512] = {'\0'}, region_buffer[512] = {'\0'};
unsigned int sip, eip, mip, ip_list[5];
int count = 0, took;
long s_time, t_time, c_time = 0;
searcher_test_t test;
for (i = 2; i < argc; i++) {
r = argv[i];
if (strlen(r) < 5) {
continue;
}
if (r[0] != '-' || r[1] != '-') {
continue;
}
if (strchr(r, '=') == NULL) {
printf("missing = for args pair '%s'\n", r);
return;
}
n = sscanf(r+2, "%32[^=]=%255[^\n]", key, val);
if (n != 2) {
printf("invalid option flag `%s`\n", r);
return;
}
if (strcmp(key, "db") == 0) {
snprintf(db_file, sizeof(db_file), "%s", val);
} else if (strcmp(key, "src") == 0) {
snprintf(src_file, sizeof(src_file), "%s", val);
} else if (strcmp(key, "cache-policy") == 0) {
memcpy(cache_policy, val, sizeof(cache_policy) - 1);
} else {
printf("undefined option `%s`\n", r);
return;
}
}
if (strlen(db_file) < 1 || strlen(src_file) < 1) {
printf("%s bench [command options]\n", argv[0]);
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --src string source ip text file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
// printf("db_file=%s, src_file=%s, cache_policy=%s\n", db_file, src_file, cache_policy);
s_time = xdb_now();
err = init_searcher_test(&test, db_file, cache_policy);
if (err != 0) {
// the init function will print the details;
return;
}
// open the source file
handle = fopen(src_file, "r");
if (handle == NULL) {
printf("failed to open source text file `%s`\n", src_file);
return;
}
while(fgets(line, sizeof(line), handle) != NULL) {
n = sscanf(line, "%15[^|]|%15[^|]|%511[^\n]", sip_str, eip_str, src_region);
if (n != 3) {
printf("invalid ip segment line `%s`\n", line);
return;
}
if (xdb_check_ip(sip_str, &sip) != 0) {
printf("invalid start ip `%s`\n", sip_str);
return;
}
if (xdb_check_ip(eip_str, &eip) != 0) {
printf("invalid end ip `%s`\n", sip_str);
return;
}
if (sip > eip) {
printf("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str);
return;
}
mip = xdb_mip(sip, eip);
ip_list[0] = sip;
ip_list[1] = xdb_mip(sip, mip);
ip_list[2] = mip;
ip_list[3] = xdb_mip(mip, eip);
ip_list[4] = eip;
for (i = 0; i < 5; i++) {
t_time = xdb_now();
err = xdb_search(&test.searcher, ip_list[i], region_buffer, sizeof(region_buffer));
if (err != 0) {
xdb_long2ip(ip_list[i], sip_str);
printf("failed to search ip `%s` with errno=%d\n", sip_str, err);
return;
}
c_time += xdb_now() - t_time;
// check the region info
if (strcmp(region_buffer, src_region) != 0) {
xdb_long2ip(ip_list[i], sip_str);
printf("failed to search(%s) with (%s != %s)\n", sip_str, region_buffer, src_region);
return;
}
count++;
}
};
took = xdb_now() - s_time;
destroy_searcher_test(&test);
fclose(handle);
printf("Bench finished, {cache_policy: %s, total: %d, took: %.3fs, cost: %d μs/op}\n",
cache_policy, count, took/1e6, count == 0 ? 0 : (int)(c_time/count));
}
int main(int argc, char *argv[]) {
if (argc < 2) {
print_help(argv);
return 0;
}
char *opt = argv[1];
if (strcmp(opt, "search") == 0) {
test_search(argc, argv);
} else if (strcmp(opt, "bench") == 0) {
test_bench(argc, argv);
} else {
print_help(argv);
}
return 0;
}
binding/c/util_test.c 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#include "stdio.h"
#include "xdb_searcher.h"
void test_check_ip() {
char *ip_list[] = {
"1.2.3.4", "192.168.2.3", "120.24.78.129", "255.255.255.0",
"256.7.12.9", "12.56.78.320", "32.12.45.192", "222.221.220.219",
"192.168.1.101 ", "132.96.12.98a", "x23.12.2.12"
};
int errcode, i;
unsigned int ip;
char ip_buff[16] = {'\0'};
for (i = 0; i < 11; i++) {
errcode = xdb_check_ip(ip_list[i], &ip);
if (errcode != 0) {
printf("invalid ip address `%s`\n", ip_list[i]);
continue;
}
xdb_long2ip(ip, ip_buff);
printf("long(%-15s)=%-10u, long2ip(%-10u)=%-15s", ip_list[i], ip, ip, ip_buff);
if (strcmp(ip_list[i], ip_buff) != 0) {
printf(" --[Failed]\n");
} else {
printf(" --[Ok]\n");
}
}
}
void test_load_header() {
xdb_header_t *header = xdb_load_header_from_file("../../data/ip2region.xdb");
if (header == NULL) {
printf("failed to load header");
} else {
printf("header loaded: {\n"
" version: %d, \n"
" index_policy: %d, \n"
" created_at: %u, \n"
" start_index_ptr: %d, \n"
" end_index_ptr: %d\n"
" length: %d\n"
"}\n",
header->version, header->index_policy, header->created_at,
header->start_index_ptr, header->end_index_ptr, header->length
);
}
xdb_close_header(header);
}
void test_load_vector_index() {
xdb_vector_index_t *v_index = xdb_load_vector_index_from_file("../../data/ip2region.xdb");
if (v_index == NULL) {
printf("failed to load vector index from file\n");
} else {
printf("vector index loaded from file, length=%d\n", v_index->length);
}
xdb_close_vector_index(v_index);
}
void test_load_content() {
xdb_content_t *content = xdb_load_content_from_file("../../data/ip2region.xdb");
if (content == NULL) {
printf("failed to load content from file\n");
} else {
printf("content loaded from file, length=%d\n", content->length);
}
xdb_close_content(content);
}
// valgrind --tool=memcheck --leak-check=full ./a.out
int main(int argc, char *argv[]) {
printf("test check ip ... \n");
test_check_ip();
printf("|--done\n\n");
printf("test load header ... \n");
test_load_header();
printf("|--done\n\n");
printf("test load vector index ... \n");
test_load_vector_index();
printf("|--done\n\n");
printf("test load content ... \n");
test_load_content();
printf("|--done\n\n");
return 0;
}
\ No newline at end of file
binding/c/xdb_searcher.c 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#include "sys/time.h"
#include "xdb_searcher.h"
// internal function prototype define
XDB_PRIVATE(int) read(xdb_searcher_t *, long offset, char *, size_t length);
XDB_PRIVATE(int) xdb_new_base(xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index, const xdb_content_t *c_buffer) {
memset(xdb, 0x00, sizeof(xdb_searcher_t));
// check the content buffer first
if (c_buffer != NULL) {
xdb->v_index = NULL;
xdb->content = c_buffer;
return 0;
}
// open the xdb binary file
FILE *handle = fopen(db_path, "r");
if (handle == NULL) {
return 1;
}
xdb->handle = handle;
xdb->v_index = v_index;
return 0;
}
// xdb searcher new api define
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_searcher_t *xdb, const char *db_path) {
return xdb_new_base(xdb, db_path, NULL, NULL);
}
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index) {
return xdb_new_base(xdb, db_path, v_index, NULL);
}
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_searcher_t *xdb, const xdb_content_t *c_buffer) {
return xdb_new_base(xdb, NULL, NULL, c_buffer);
}
XDB_PUBLIC(void) xdb_close(void *ptr) {
xdb_searcher_t *xdb = (xdb_searcher_t *) ptr;
if (xdb->handle != NULL) {
fclose(xdb->handle);
xdb->handle = NULL;
}
}
// --- xdb searcher search api define
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *xdb, const char *str_ip, char *region_buffer, size_t length) {
unsigned int ip = 0;
int errcode = xdb_check_ip(str_ip, &ip);
if (errcode != 0) {
return 10 + errcode;
} else {
return xdb_search(xdb, ip, region_buffer, length);
}
}
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *xdb, unsigned int ip, char *region_buffer, size_t length) {
int il0, il1, idx, err, l, h, m, data_len;
unsigned int s_ptr, e_ptr, p, sip, eip, data_ptr;
char vector_buffer[xdb_vector_index_size], segment_buffer[xdb_segment_index_size];
// reset the io counter
xdb->io_count = 0;
// locate the segment index block based on the vector index
il0 = ((int) (ip >> 24)) & 0xFF;
il1 = ((int) (ip >> 16)) & 0xFF;
idx = il0 * xdb_vector_index_cols * xdb_vector_index_size + il1 * xdb_vector_index_size;
if (xdb->v_index != NULL) {
s_ptr = xdb_get_uint(xdb->v_index->buffer, idx);
e_ptr = xdb_get_uint(xdb->v_index->buffer, idx + 4);
} else if (xdb->content != NULL) {
s_ptr = xdb_get_uint(xdb->content->buffer, xdb_header_info_length + idx);
e_ptr = xdb_get_uint(xdb->content->buffer, xdb_header_info_length + idx + 4);
} else {
err = read(xdb, xdb_header_info_length + idx, vector_buffer, sizeof(vector_buffer));
if (err != 0) {
return 10 + err;
}
s_ptr = xdb_get_uint(vector_buffer, 0);
e_ptr = xdb_get_uint(vector_buffer, 4);
}
// printf("s_ptr=%u, e_ptr=%u\n", s_ptr, e_ptr);
// binary search to get the final region info
data_len = 0, data_ptr = 0;
l = 0, h = ((int) (e_ptr - s_ptr)) / xdb_segment_index_size;
while (l <= h) {
m = (l + h) >> 1;
p = s_ptr + m * xdb_segment_index_size;
// read the segment index item
err = read(xdb, p, segment_buffer, sizeof(segment_buffer));
if (err != 0) {
return 20 + err;
}
// decode the data fields as needed
sip = xdb_get_uint(segment_buffer, 0);
if (ip < sip) {
h = m - 1;
} else {
eip = xdb_get_uint(segment_buffer, 4);
if (ip > eip) {
l = m + 1;
} else {
data_len = xdb_get_ushort(segment_buffer, 8);
data_ptr = xdb_get_uint(segment_buffer, 10);
break;
}
}
}
// printf("data_len=%u, data_ptr=%u\n", data_len, data_ptr);
if (data_len == 0) {
region_buffer[0] = '\0';
return 0;
}
// buffer length checking
if (data_len >= length) {
return 1;
}
err = read(xdb, data_ptr, region_buffer, data_len);
if (err != 0) {
return 30 + err;
}
// auto append a NULL-end
region_buffer[data_len] = '\0';
return 0;
}
XDB_PRIVATE(int) read(xdb_searcher_t *xdb, long offset, char *buffer, size_t length) {
// check the xdb content cache first
if (xdb->content != NULL) {
memcpy(buffer, xdb->content->buffer + offset, length);
return 0;
}
// seek to the offset
if (fseek(xdb->handle, offset, SEEK_SET) == -1) {
return 1;
}
xdb->io_count++;
if (fread(buffer, 1, length, xdb->handle) != length) {
return 2;
}
return 0;
}
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *xdb) {
return xdb->io_count;
}
// --- buffer load util functions
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *handle) {
xdb_header_t *header;
unsigned int size = xdb_header_info_length;
// entry alloc
header = (xdb_header_t *) xdb_malloc(sizeof(xdb_header_t));
if (header == NULL) {
return NULL;
}
if (fseek(handle, 0, SEEK_SET) == -1) {
xdb_free(header);
return NULL;
}
if (fread(header->buffer, 1,size, handle) != size) {
xdb_free(header);
return NULL;
}
// fill the fields
header->length = size;
header->version = (unsigned short) xdb_get_ushort(header->buffer, 0);
header->index_policy = (unsigned short) xdb_get_ushort(header->buffer, 2);
header->created_at = xdb_get_uint(header->buffer, 4);
header->start_index_ptr = xdb_get_uint(header->buffer, 8);
header->end_index_ptr = xdb_get_uint(header->buffer,12);
return header;
}
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *db_path) {
xdb_header_t *header;
FILE *handle = fopen(db_path, "r");
if (handle == NULL) {
return NULL;
}
header = xdb_load_header(handle);
fclose(handle);
return header;
}
XDB_PUBLIC(void) xdb_close_header(void *ptr) {
xdb_header_t *header = (xdb_header_t *) ptr;
if (header->length > 0) {
header->length = 0;
xdb_free(header);
}
}
// --- vector index
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *handle) {
xdb_vector_index_t *v_index;
unsigned int size = xdb_vector_index_length;
// seek to the vector index offset
if (fseek(handle, xdb_header_info_length, SEEK_SET) == -1) {
return NULL;
}
// do the buffer read
v_index = (xdb_vector_index_t *) xdb_malloc(sizeof(xdb_vector_index_t));
if (v_index == NULL) {
return NULL;
}
v_index->length = size;
if (fread(v_index->buffer, 1, size, handle) != size) {
xdb_free(v_index);
return NULL;
}
return v_index;
}
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *db_path) {
xdb_vector_index_t *v_index;
FILE *handle = fopen(db_path, "r");
if (handle == NULL) {
return NULL;
}
v_index = xdb_load_vector_index(handle);
fclose(handle);
return v_index;
}
XDB_PUBLIC(void) xdb_close_vector_index(void *ptr) {
xdb_vector_index_t *v_index = (xdb_vector_index_t *) ptr;
if (v_index->length > 0) {
v_index->length = 0;
xdb_free(v_index);
}
}
// --- content buffer
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *handle) {
unsigned int size;
xdb_content_t *content;
char *ptr;
// determine the file size
if (fseek(handle, 0, SEEK_END) == -1) {
return NULL;
}
size = (unsigned int) ftell(handle);
if (fseek(handle, 0, SEEK_SET) == -1) {
return NULL;
}
// do the file read
content = (xdb_content_t *) xdb_malloc(sizeof(xdb_content_t));
if (content == NULL) {
return NULL;
}
// do the buffer alloc
content->buffer = (char *) xdb_malloc(size);
if (content->buffer == NULL) {
xdb_free(content);
return NULL;
}
// read the content into the buffer
content->length = size;
if (fread(content->buffer, 1, size, handle) != size) {
xdb_free(ptr);
return NULL;
}
return content;
}
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *db_path) {
xdb_content_t *content;
FILE *handle = fopen(db_path, "r");
if (handle == NULL) {
return NULL;
}
content = xdb_load_content(handle);
fclose(handle);
return content;
}
XDB_PUBLIC(void) xdb_close_content(void *ptr) {
xdb_content_t *content = (xdb_content_t *) ptr;
if (content->length > 0) {
content->length = 0;
xdb_free(content->buffer);
content->buffer = NULL;
xdb_free(content);
}
}
// --- End
// get unsigned long (4bytes) from a specified buffer start from the specified offset
XDB_PUBLIC(unsigned int) xdb_get_uint(const char *buffer, int offset) {
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00) |
((buffer[offset+2] << 16) & 0x00FF0000) |
((buffer[offset+3] << 24) & 0xFF000000)
);
}
// get unsigned short (2bytes) from a specified buffer start from the specified offset
XDB_PUBLIC(int) xdb_get_ushort(const char *buffer, int offset) {
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00)
);
}
// string ip to unsigned int
static int shiftIndex[4] = {24, 16, 8, 0};
XDB_PUBLIC(int) xdb_check_ip(const char *src_ip, unsigned int *dst_ip) {
char c;
int i, n, ip = 0;
const char *ptr = src_ip;
for (i = 0; i < 4; i++) {
n = 0;
while (1) {
c = *ptr;
ptr++;
if (c >= '0' && c <= '9') {
n *= 10;
n += c - '0';
} else if ((i < 3 && c == '.') || i == 3) {
// stopping at the '.' but ignore the tailing chars
// after the 3rd one (auto clean the tailing none-integer ?).
break;
} else {
return 1;
}
}
if (n > 0xFF) {
return 2;
}
ip |= (n << shiftIndex[i]);
}
*dst_ip = ip;
return 0;
}
// unsigned int ip to string ip
XDB_PUBLIC(void) xdb_long2ip(unsigned int ip, char *buffer) {
sprintf(buffer, "%d.%d.%d.%d", (ip >> 24) & 0xFF, (ip >> 16) & 0xFF, (ip >> 8) & 0xFF, ip & 0xFF);
}
// get the middle ip of a and b
XDB_PUBLIC(unsigned int) xdb_mip(unsigned long a, unsigned long b) {
return (unsigned int) ((a + b) >> 1);
}
XDB_PUBLIC(long) xdb_now() {
struct timeval c_time;
gettimeofday(&c_time, NULL);
return c_time.tv_sec * (int)1e6 + c_time.tv_usec;
}
binding/c/xdb_searcher.h 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#ifndef C_XDB_SEARCHER_H
#define C_XDB_SEARCHER_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#if ( defined(WIN32) || defined(_WIN32) || defined(__WINDOWS_) || defined(WINNT) )
# define XDB_PUBLIC(type) extern __declspec(dllexport) type
# define XDB_PRIVATE(type) static type
# define XDB_WINDOWS
#elif ( defined(linux) || defined(_UNIX) )
# define XDB_PUBLIC(type) extern type
# define XDB_PRIVATE(type) static inline type
# define XDB_LINUX
#endif
#define xdb_calloc( _blocks, _bytes ) calloc( _blocks, _bytes )
#define xdb_malloc( _bytes ) malloc( _bytes )
#define xdb_free( _ptr ) free( _ptr )
// public constants define
#define xdb_header_info_length 256
#define xdb_vector_index_rows 256
#define xdb_vector_index_cols 256
#define xdb_vector_index_size 8
#define xdb_segment_index_size 14
// cache of vector_index_row × vector_index_rows × vector_index_size
#define xdb_vector_index_length 524288
// --- buffer load util functions
// use the following buffer struct to wrap the binary buffer data
// since the buffer data could not be operated with the string API.
struct xdb_header {
unsigned short version;
unsigned short index_policy;
unsigned int created_at;
unsigned int start_index_ptr;
unsigned int end_index_ptr;
// the original buffer
unsigned int length;
char buffer[xdb_header_info_length];
};
typedef struct xdb_header xdb_header_t;
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *);
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *);
XDB_PUBLIC(void) xdb_close_header(void *);
// --- vector index buffer
struct xdb_vector_index {
unsigned int length;
char buffer[xdb_vector_index_length];
};
typedef struct xdb_vector_index xdb_vector_index_t;
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *);
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *);
XDB_PUBLIC(void) xdb_close_vector_index(void *);
// --- content buffer
struct xdb_content {
unsigned int length;
char *buffer;
};
typedef struct xdb_content xdb_content_t;
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *);
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *);
XDB_PUBLIC(void) xdb_close_content(void *);
// --- End buffer load
// xdb searcher structure
struct xdb_searcher_entry {
FILE *handle;
// header info
const char *header;
int io_count;
// vector index buffer cache.
// preload the vector index will reduce the number of IO operations
// thus speedup the search process.
const xdb_vector_index_t *v_index;
// content buffer.
// cache the whole xdb content.
const xdb_content_t *content;
};
typedef struct xdb_searcher_entry xdb_searcher_t;
// xdb searcher new api define
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_searcher_t *, const char *);
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_searcher_t *, const char *, const xdb_vector_index_t *);
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_searcher_t *, const xdb_content_t *);
XDB_PUBLIC(void) xdb_close(void *);
// xdb searcher search api define
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *, const char *, char *, size_t);
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *, unsigned int, char *, size_t);
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *);
// get unsigned long (4bytes) from a specified buffer start from the specified offset with little-endian
XDB_PUBLIC(unsigned int) xdb_get_uint(const char *, int);
// get unsigned short (2bytes) from a specified buffer start from the specified offset with little-endian
XDB_PUBLIC(int) xdb_get_ushort(const char *, int);
// check the specified string ip and convert it to an unsigned int
XDB_PUBLIC(int) xdb_check_ip(const char *, unsigned int *);
// unsigned int ip to string ip
XDB_PUBLIC(void) xdb_long2ip(unsigned int, char *);
// get the middle ip of a and b
XDB_PUBLIC(unsigned int) xdb_mip(unsigned long, unsigned long);
// get the current time in microseconds
XDB_PUBLIC(long) xdb_now();
#endif //C_XDB_SEARCHER_H
binding/csharp/.gitignore 0 → 100644
浏览文件 @ 65edd65b
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUNIT
*.VisualState.xml
TestResult.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# JustCode is a .NET coding add-in
.JustCode
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!?*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
*- Backup*.rdl
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# JetBrains Rider
.idea/
*.sln.iml
# CodeRush personal settings
.cr/personal
# Python Tools for Visual Studio (PTVS)
__pycache__/
*.pyc
# Cake - Uncomment if you are using it
# tools/**
# !tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
.mfractor/
# Local History for Visual Studio
.localhistory/
\ No newline at end of file
binding/csharp/IP2Region.SearchTest/App.config 0 → 100644
浏览文件 @ 65edd65b
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
</startup>
</configuration>
\ No newline at end of file
binding/csharp/IP2Region.SearchTest/IP2Region.SearchTest.csproj 0 → 100644
浏览文件 @ 65edd65b
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<ProjectGuid>{FF136AEA-D41F-4C14-842B-CE597BB97916}</ProjectGuid>
<OutputType>Exe</OutputType>
<RootNamespace>IP2Region.SearchTest</RootNamespace>
<AssemblyName>IP2Region.SearchTest</AssemblyName>
<TargetFrameworkVersion>v4.7.2</TargetFrameworkVersion>
<FileAlignment>512</FileAlignment>
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
<Deterministic>true</Deterministic>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugSymbols>true</DebugSymbols>
<DebugType>full</DebugType>
<Optimize>false</Optimize>
<OutputPath>bin\Debug\</OutputPath>
<DefineConstants>DEBUG;TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<Optimize>true</Optimize>
<OutputPath>bin\Release\</OutputPath>
<DefineConstants>TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
<ItemGroup>
<Reference Include="System" />
<Reference Include="System.Core" />
<Reference Include="System.Xml.Linq" />
<Reference Include="System.Data.DataSetExtensions" />
<Reference Include="Microsoft.CSharp" />
<Reference Include="System.Data" />
<Reference Include="System.Net.Http" />
<Reference Include="System.Xml" />
</ItemGroup>
<ItemGroup>
<Compile Include="Program.cs" />
<Compile Include="Properties\AssemblyInfo.cs" />
</ItemGroup>
<ItemGroup>
<None Include="App.config" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region\IP2Region.csproj">
<Project>{483575a5-ffb3-4131-a2dc-6d3ad3bb268f}</Project>
<Name>IP2Region</Name>
</ProjectReference>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
</Project>
\ No newline at end of file
binding/csharp/IP2Region.SearchTest/Program.cs 0 → 100644
浏览文件 @ 65edd65b
using IP2Region.xdb;
using System;
using System.Diagnostics;
using System.IO;
namespace IP2Region.SearchTest
{
internal class Program
{
public static void PrintHelp(String[] args)
{
Console.WriteLine("ip2region xdb searcher");
Console.WriteLine("IP2Region.SearchTest.exe [command] [command options]");
Console.WriteLine("Command: ");
Console.WriteLine(" search search input test");
Console.WriteLine(" bench search bench test");
}
public static Searcher CreateSearcher(String dbPath, String cachePolicy)
{
if ("file" == cachePolicy)
{
return Searcher.NewWithFileOnly(dbPath);
}
else if ("vectorIndex" == cachePolicy)
{
byte[] vIndex = Searcher.LoadVectorIndexFromFile(dbPath);
return Searcher.NewWithVectorIndex(dbPath, vIndex);
}
else if ("content" == cachePolicy)
{
byte[] cBuff = Searcher.LoadContentFromFile(dbPath);
return Searcher.NewWithBuffer(cBuff);
}
else
{
throw new Exception("invalid cache policy `" + cachePolicy + "`, options: file/vectorIndex/content");
}
}
public static void SearchTest(string[] args)
{
String dbPath = "", cachePolicy = "vectorIndex";
foreach (string r in args)
{
if (r.Length < 5)
{
continue;
}
if (r.IndexOf("--") != 0)
{
continue;
}
int sIdx = r.IndexOf('=');
if (sIdx < 0)
{
Console.WriteLine("missing = for args pair `{0}`", r);
return;
}
String key = r.Substring(2, sIdx - 2);
String val = r.Substring(sIdx + 1);
// Console.WriteLinef("key=%s, val=%s\n", key, val);
if ("db" == key)
{
dbPath = val;
}
else if ("cache-policy" == key)
{
cachePolicy = val;
}
else
{
Console.WriteLine("undefined option `{0}`", r);
return;
}
}
if (dbPath.Length < 1)
{
Console.WriteLine("IP2Region.SearchTest.exe search [command options]");
Console.WriteLine("options:");
Console.WriteLine(" --db string ip2region binary xdb file path");
Console.WriteLine(" --cache-policy string cache policy: file/vectorIndex/content");
return;
}
Searcher searcher = CreateSearcher(dbPath, cachePolicy);
Console.WriteLine("ip2region xdb searcher test program, cachePolicy: {0}\ntype 'quit' to exit", cachePolicy);
while (true)
{
Console.Write("ip2region>> ");
var line = Console.ReadLine().Trim();
if (line.Length < 2) continue;
if (line == "quit") break;
try
{
var st = new Stopwatch();
st.Start();
var region = searcher.Search(line);
st.Stop();
var cost = st.ElapsedMilliseconds;
Console.WriteLine("{{region: {0}, ioCount: {1}, took: {2} ms}}", region, searcher.IOCount, cost);
}
catch (Exception e)
{
Console.WriteLine("{{err:{0}, ioCount: {1}}}", e, searcher.IOCount);
}
}
Console.WriteLine("searcher test program exited, thanks for trying");
}
public static void BenchTest(String[] args)
{
String dbPath = "", srcPath = "", cachePolicy = "vectorIndex";
foreach (String r in args)
{
if (r.Length < 5)
{
continue;
}
if (r.IndexOf("--") != 0)
{
continue;
}
int sIdx = r.IndexOf('=');
if (sIdx < 0)
{
Console.WriteLine("missing = for args pair `{0}`", r);
return;
}
String key = r.Substring(2, sIdx - 2);
String val = r.Substring(sIdx + 1);
if ("db" == key)
{
dbPath = val;
}
else if ("src" == key)
{
srcPath = val;
}
else if ("cache-policy" == key)
{
cachePolicy = val;
}
else
{
Console.WriteLine("undefined option `{0}`", r);
return;
}
}
if (dbPath.Length < 1 || srcPath.Length < 1)
{
Console.WriteLine("IP2Region.SearchTest.exe bench [command options]");
Console.WriteLine("options:");
Console.WriteLine(" --db string ip2region binary xdb file path");
Console.WriteLine(" --src string source ip text file path");
Console.WriteLine(" --cache-policy string cache policy: file/vectorIndex/content");
return;
}
Searcher searcher = CreateSearcher(dbPath, cachePolicy);
long count = 0;
var sw = new Stopwatch();
var lines = File.ReadAllLines(srcPath);
foreach (var line in lines)
{
String l = line.Trim();
String[] ps = l.Split(new[] { '|' }, 3);
if (ps.Length != 3)
{
Console.WriteLine("invalid ip segment `{0}`", l);
return;
}
long sip;
try
{
sip = Searcher.checkIP(ps[0]);
}
catch (Exception e)
{
Console.WriteLine("check start ip `{0}`: {1}", ps[0], e);
return;
}
long eip;
try
{
eip = Searcher.checkIP(ps[1]);
}
catch (Exception e)
{
Console.WriteLine("check end ip `{0}`: {1}", ps[1], e);
return;
}
if (sip > eip)
{
Console.WriteLine("start ip({0}) should not be greater than end ip({1})", ps[0], ps[1]);
return;
}
long mip = (sip + eip) >> 1;
foreach (var ip in new long[] { sip, (sip + mip) >> 1, mip, (mip + eip) >> 1, eip })
{
sw.Start();
String region = searcher.Search(ip);
sw.Stop();
// check the region info
if (ps[2] != (region))
{
Console.WriteLine("failed search({0}) with ({1} != {2})\n", Searcher.Long2ip(ip), region, ps[2]);
return;
}
count++;
}
}
Console.WriteLine("Bench finished, {{cachePolicy: {0}, total: {1}, took: {2}, cost: {3} ms/op}}",
cachePolicy, count, sw.Elapsed,
count == 0 ? 0 : sw.ElapsedMilliseconds / count);
}
static void Main(string[] args)
{
if (args.Length < 1)
{
PrintHelp(args);
return;
}
switch (args[0])
{
case "search":
try
{
SearchTest(args);
}
catch (Exception e)
{
Console.WriteLine("failed running search test: {0}", e);
}
break;
case "bench":
try
{
BenchTest(args);
}
catch (Exception e)
{
Console.WriteLine("failed running bench test: {0}", e);
}
break;
default: PrintHelp(args); break;
}
}
}
}
binding/csharp/IP2Region.SearchTest/Properties/AssemblyInfo.cs 0 → 100644
浏览文件 @ 65edd65b
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
// 有关程序集的一般信息由以下
// 控制。更改这些特性值可修改
// 与程序集关联的信息。
[assembly: AssemblyTitle("IP2Region.SearchTest")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("IP2Region.SearchTest")]
[assembly: AssemblyCopyright("Copyright © 2022")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// 将 ComVisible 设置为 false 会使此程序集中的类型
//对 COM 组件不可见。如果需要从 COM 访问此程序集中的类型
//请将此类型的 ComVisible 特性设置为 true。
[assembly: ComVisible(false)]
// 如果此项目向 COM 公开，则下列 GUID 用于类型库的 ID
[assembly: Guid("ff136aea-d41f-4c14-842b-ce597bb97916")]
// 程序集的版本信息由下列四个值组成:
//
// 主版本
// 次版本
// 生成号
// 修订号
//
//可以指定所有这些值，也可以使用“生成号”和“修订号”的默认值
//通过使用 "*"，如下所示:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]
binding/csharp/IP2Region.sln 0 → 100644
浏览文件 @ 65edd65b

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 17
VisualStudioVersion = 17.1.32228.430
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "IP2Region", "IP2Region\IP2Region.csproj", "{483575A5-FFB3-4131-A2DC-6D3AD3BB268F}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "IP2Region.SearchTest", "IP2Region.SearchTest\IP2Region.SearchTest.csproj", "{FF136AEA-D41F-4C14-842B-CE597BB97916}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{483575A5-FFB3-4131-A2DC-6D3AD3BB268F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{483575A5-FFB3-4131-A2DC-6D3AD3BB268F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{483575A5-FFB3-4131-A2DC-6D3AD3BB268F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{483575A5-FFB3-4131-A2DC-6D3AD3BB268F}.Release|Any CPU.Build.0 = Release|Any CPU
{FF136AEA-D41F-4C14-842B-CE597BB97916}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{FF136AEA-D41F-4C14-842B-CE597BB97916}.Debug|Any CPU.Build.0 = Debug|Any CPU
{FF136AEA-D41F-4C14-842B-CE597BB97916}.Release|Any CPU.ActiveCfg = Release|Any CPU
{FF136AEA-D41F-4C14-842B-CE597BB97916}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {6AEA440B-0D90-4F6E-BA9A-D42B27A78D7B}
EndGlobalSection
EndGlobal
binding/csharp/IP2Region/IP2Region.csproj 0 → 100644
浏览文件 @ 65edd65b
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFrameworks>netstandard2.0;netcoreapp3.1;net5.0;net6.0</TargetFrameworks>
</PropertyGroup>
</Project>
binding/csharp/IP2Region/xdb/Header.cs 0 → 100644
浏览文件 @ 65edd65b
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace IP2Region.xdb
{
public class Header
{
public int Version { get; }
public int IndexPolicy { get; }
public int CreatedAt { get; }
public int StartIndexPtr { get; }
public int EndIndexPtr { get; }
public byte[] Buffer { get; }
public Header(byte[] buff)
{
if (buff == null) throw new ArgumentNullException(nameof(buff));
if (buff.Length < 16) throw new ArgumentOutOfRangeException(nameof(buff));
Version = Searcher.GetInt2(buff, 0);
IndexPolicy = Searcher.GetInt2(buff, 2);
CreatedAt = Searcher.GetInt(buff, 4);
StartIndexPtr = Searcher.GetInt(buff, 8);
EndIndexPtr = Searcher.GetInt(buff, 12);
Buffer = buff;
}
public override string ToString()
{
return "{" +
"Version: " + Version + ',' +
"IndexPolicy: " + IndexPolicy + ',' +
"CreatedAt: " + CreatedAt + ',' +
"StartIndexPtr: " + StartIndexPtr + ',' +
"EndIndexPtr: " + EndIndexPtr +
'}';
}
}
}
binding/csharp/IP2Region/xdb/Searcher.cs 0 → 100644
浏览文件 @ 65edd65b
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace IP2Region.xdb
{
public class Searcher
{
// constant defined copied from the xdb maker
public static int HeaderInfoLength = 256;
public static int VectorIndexRows = 256;
public static int VectorIndexCols = 256;
public static int VectorIndexSize = 8;
public static int SegmentIndexSize = 14;
// random access file handle for file based search
private readonly Stream handle;
private int ioCount = 0;
public int IOCount => this.ioCount;
// vector index.
// use the byte[] instead of VectorIndex entry array to keep
// the minimal memory allocation.
private readonly byte[] vectorIndex;
// xdb content buffer, used for in-memory search
private readonly byte[] contentBuff;
// --- static method to create searchers
public static Searcher NewWithFileOnly(String dbPath)
{
return new Searcher(dbPath, null, null);
}
public static Searcher NewWithVectorIndex(String dbPath, byte[] vectorIndex)
{
return new Searcher(dbPath, vectorIndex, null);
}
public static Searcher NewWithBuffer(byte[] cBuff)
{
return new Searcher(null, null, cBuff);
}
// --- End of creator
public Searcher(string dbFile, byte[] vectorIndex, byte[] cBuff)
{
if (cBuff != null)
{
this.handle = null;
this.vectorIndex = null;
this.contentBuff = cBuff;
}
else
{
this.handle = File.OpenRead(dbFile);
this.vectorIndex = vectorIndex;
this.contentBuff = null;
}
}
public void Close()
{
if (this.handle != null) this.handle.Close();
}
public string Search(string ipStr)
{
var ip = checkIP(ipStr);
return Search(ip);
}
public string Search(long ip)
{
// reset the global counter
this.ioCount = 0;
// locate the segment index block based on the vector index
int sPtr = 0, ePtr = 0;
int il0 = (int)((ip >> 24) & 0xFF);
int il1 = (int)((ip >> 16) & 0xFF);
int idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
if (vectorIndex != null)
{
sPtr = GetInt(vectorIndex, idx);
ePtr = GetInt(vectorIndex, idx + 4);
}
else if (contentBuff != null)
{
sPtr = GetInt(contentBuff, HeaderInfoLength + idx);
ePtr = GetInt(contentBuff, HeaderInfoLength + idx + 4);
}
else
{
byte[] vectorBuff = new byte[VectorIndexSize];
Read(HeaderInfoLength + idx, vectorBuff);
sPtr = GetInt(vectorBuff, 0);
ePtr = GetInt(vectorBuff, 4);
}
// binary search the segment index block to get the region info
byte[] buff = new byte[SegmentIndexSize];
int dataLen = -1, dataPtr = -1;
int l = 0, h = (ePtr - sPtr) / SegmentIndexSize;
while (l <= h)
{
int m = (l + h) >> 1;
int p = sPtr + m * SegmentIndexSize;
// read the segment index
Read(p, buff);
long sip = GetIntLong(buff, 0);
if (ip < sip)
{
h = m - 1;
}
else
{
long eip = GetIntLong(buff, 4);
if (ip > eip)
{
l = m + 1;
}
else
{
dataLen = GetInt2(buff, 8);
dataPtr = GetInt(buff, 10);
break;
}
}
}
// empty match interception
// System.out.printf("dataLen: %d, dataPtr: %d\n", dataLen, dataPtr);
if (dataPtr < 0) return null;
// load and return the region data
byte[] regionBuff = new byte[dataLen];
Read(dataPtr, regionBuff);
//return new String(regionBuff, "utf-8");
return Encoding.UTF8.GetString(regionBuff);
}
protected virtual void Read(int offset, byte[] buffer)
{
// check the in-memory buffer first
if (contentBuff != null)
{
// @TODO: reduce data copying, directly decode the data ?
//System.arraycopy(contentBuff, offset, buffer, 0, buffer.length);
Array.Copy(contentBuff, offset, buffer, 0, buffer.Length);
return;
}
// read from the file handle
if (handle == null) throw new ArgumentNullException(nameof(handle));
handle.Seek(offset, SeekOrigin.Begin);
this.ioCount++;
var rLen = handle.Read(buffer, 0, buffer.Length);
if (rLen != buffer.Length) throw new IOException("incomplete read: read bytes should be " + buffer.Length);
}
// --- static cache util function
public static Header LoadHeader(Stream stream)
{
stream.Seek(0, SeekOrigin.Begin);
var buffer = new byte[HeaderInfoLength];
stream.Read(buffer, 0, HeaderInfoLength);
return new Header(buffer);
}
public static Header LoadHeaderFromFile(string dbPath)
{
using (var fs = File.OpenRead(dbPath)) return LoadHeader(fs);
}
public static byte[] LoadVectorIndex(Stream stream)
{
stream.Seek(HeaderInfoLength, SeekOrigin.Begin);
int len = VectorIndexRows * VectorIndexCols * SegmentIndexSize;
var buff = new byte[len];
var rLen = stream.Read(buff, 0, buff.Length);
if (rLen != len) throw new IOException("incomplete read: read bytes should be " + len);
return buff;
}
public static byte[] LoadVectorIndexFromFile(string dbPath)
{
using (var fs = File.OpenRead(dbPath)) return LoadVectorIndex(fs);
}
public static byte[] LoadContent(Stream stream)
{
stream.Seek(0, SeekOrigin.Begin);
using (var ms = new MemoryStream())
{
stream.CopyTo(ms);
return ms.ToArray();
}
}
public static byte[] LoadContentFromFile(string dbPath)
{
using (var fs = File.OpenRead(dbPath)) return LoadContent(fs);
}
public static int GetInt2(byte[] b, int offset)
{
return (
(b[offset++] & 0x000000FF) |
(b[offset] & 0x0000FF00)
);
}
public static int GetInt(byte[] b, int offset)
{
return (
((b[offset++]) & 0x000000FF) |
((b[offset++] << 8) & 0x0000FF00) |
((b[offset++] << 16) & 0x00FF0000) |
(int)((b[offset] << 24) & 0xFF000000)
);
}
public static long GetIntLong(byte[] b, int offset)
{
return (
((b[offset++] & 0x000000FFL)) |
((b[offset++] << 8) & 0x0000FF00L) |
((b[offset++] << 16) & 0x00FF0000L) |
((b[offset] << 24) & 0xFF000000L)
);
}
/* long int to ip string */
public static string Long2ip(long ip)
{
return string.Join(".", (ip >> 24) & 0xFF, (ip >> 16) & 0xFF, (ip >> 8) & 0xFF, (ip) & 0xFF);
}
public static byte[] shiftIndex = { 24, 16, 8, 0 };
/* check the specified ip address */
public static long checkIP(String ip)
{
String[]
ps = ip.Split('.');
if (ps.Length != 4) throw new Exception("invalid ip address `" + ip + "`");
long ipDst = 0;
for (int i = 0; i < ps.Length; i++)
{
int val = Convert.ToInt32(ps[i]);
if (val > 255)
{
throw new Exception("ip part `" + ps[i] + "` should be less then 256");
}
ipDst |= ((long)val << shiftIndex[i]);
}
return ipDst & 0xFFFFFFFFL;
}
}
}
binding/csharp/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# ip2region csharp 查询客户端实现
# 使用方式
### 引入文件
将 IP2Region\xdb\Header.cs 和 IP2Region\xdb\Searcher.cs 复制到项目文件中即可。
兼容框架：netstandard2.0、netcoreapp3.1、net5.0和net6.0。其余框架请自行测试。
### 完全基于文件的查询
```csharp
// 1、创建 searcher 对象
String dbPath = "ip2region.xdb file path";
Searcher searcher = null;
try
{
searcher = Searcher.NewWithFileOnly(dbPath);
}
catch (IOException e)
{
Console.WriteLine("failed to create searcher with `{0}`: {1}", dbPath, e);
return;
}
// 2、查询
String ip = "1.2.3.4";
try
{
String region = searcher.Search(ip);
Console.WriteLine("{{region: {0}, ioCount: {1}}}\n", region, searcher.IOCount);
}
catch (Exception e)
{
Console.WriteLine("failed to search({0}): {1}\n", ip, e);
}
// 3、备注：并发使用，每个线程需要创建一个独立的 searcher 对象单独使用。
```
### 缓存 `VectorIndex` 索引
```csharp
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 中预先加载 VectorIndex 缓存，并且把这个得到的数据作为全局变量，后续反复使用。
byte[] vIndex;
try
{
vIndex = Searcher.LoadVectorIndexFromFile(dbPath);
}
catch (Exception e)
{
Console.WriteLine("failed to load vector index from `{0}`: {1}", dbPath, e);
return;
}
// 2、使用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
Searcher searcher;
try
{
searcher = Searcher.NewWithVectorIndex(dbPath, vIndex);
}
catch (Exception e)
{
Console.WriteLine("failed to create vectorIndex cached searcher with `{0}`: {1}", dbPath, e);
return;
}
// 3、查询
String ip = "1.2.3.4";
try
{
String region = searcher.Search(ip);
Console.WriteLine("{{region: {0}, ioCount: {1}}}", region, searcher.IOCount);
}
catch (Exception e)
{
Console.WriteLine("failed to search({0}): {1}", ip, e);
}
// 备注：每个线程需要单独创建一个独立的 Searcher 对象，但是都共享全局的制度 vIndex 缓存。
```
### 缓存整个 `xdb` 数据
我们也可以预先加载整个 ip2region.xdb 的数据到内存，然后基于这个数据创建查询对象来实现完全基于文件的查询，类似之前的 memory search。
```csharp
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 加载整个 xdb 到内存。
byte[] cBuff;
try
{
cBuff = Searcher.LoadContentFromFile(dbPath);
}
catch (Exception e)
{
Console.WriteLine("failed to load content from `{0}`: {1}", dbPath, e);
return;
}
// 2、使用上述的 cBuff 创建一个完全基于内存的查询对象。
Searcher searcher;
try
{
searcher = Searcher.NewWithBuffer(cBuff);
}
catch (Exception e)
{
Console.WriteLine("failed to create content cached searcher: {0}", e);
return;
}
// 3、查询
String ip = "1.2.3.4";
try {
String region = searcher.Search(ip);
Console.WriteLine("{{region: {0}, ioCount: {1}}}", region, searcher.IOCount);
}
catch (Exception e)
{
Console.WriteLine("failed to search({0}): {1}", ip, e);
}
// 备注：并发使用，用整个 xdb 数据缓存创建的查询对象可以安全的用于并发，也就是你可以把这个 searcher 对象做成全局对象去跨线程访问。
```
# 编译测试程序
使用 Visual Stuido 打开 IP2Region 解决方案，按“F6”生成测试程序。然后会在 `IP2Region.SearchTest\bin\Debug` 目录下得到一个 `IP2Region.SearchTest.exe` 测试程序。
# 查询测试
可以通过 `IP2Region.SearchTest.exe search` 命令来测试查询：
```bash
IP2Region.SearchTest.exe search
IP2Region.SearchTest.exe search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 文件进行查询测试：
```bash
IP2Region.SearchTest.exe search --db=../../../../../data/ip2region.xdb
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, ioCount: 7, took: 0 ms}
ip2region>>
```
输入 ip 即可进行查询测试，也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的查询效果。
# bench 测试
可以通过 `IP2Region.SearchTest.exe bench` 命令来进行 bench 测试，一方面确保 `xdb` 文件没有错误，一方面可以评估查询性能：
```bash
IP2Region.SearchTest.exe bench
IP2Region.SearchTest.exe bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 文件进行 bench 测试：
```bash
IP2Region.SearchTest.exe bench --db=../../../../../data/ip2region.xdb --src=../../../../../data/ip.merge.txt
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 00:00:48.0082981, cost: 0 ms/op}
```
可以通过分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效果。
@Note: 注意 bench 使用的 src 文件要是生成对应 xdb 文件相同的源文件。
binding/golang/Makefile 0 → 100644
浏览文件 @ 65edd65b
# ip2region golang binding makefile
all: build
.PHONY: all
build:
go build -o xdb_searcher
test:
go test -v ./...
clean:
find ./ -name xdb_searcher | xargs rm -f
\ No newline at end of file
binding/golang/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# ip2region xdb golang 查询客户端实现
# 使用方式
### package 获取
```bash
go get github.com/lionsoul2014/ip2region/binding/golang
```
### 完全基于文件的查询
```golang
import (
"fmt"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"time"
)
func main() {
var dbPath = "ip2region.xdb file path"
searcher, err := xdb.NewWithFileOnly(dbPath)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer searcher.Close()
// do the search
var ip = "1.2.3.4"
var tStart = time.Now()
region, err := searcher.SearchByStr(ip)
if err != nil {
fmt.Printf("failed to SearchIP(%s): %s\n", ip, err)
return
}
fmt.Printf("{region: %s, took: %s}\n", region, time.Since(tStart))
// 备注：并发使用，每个 goroutine 需要创建一个独立的 searcher 对象。
}
```
### 缓存 `VetorIndex` 索引
可以预先加载 `vecotorIndex` 缓存，然后做成全局变量，每次创建 searcher 的时候使用全局的 `vectorIndex`，可以减少一次固定的 IO 操作从而加速查询，减少系统 io 压力。
```golang
// 1、从 dbPath 加载 VectorIndex 缓存，把下述 vIndex 变量全局到内存里面。
vIndex, err := LoadVectorIndexFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load vector index from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
searcher, err := xdb.NewWithVectorIndex(dbPath, vIndex)
if err != nil {
fmt.Printf("failed to create searcher with vector index: %s\n", err)
return
}
// 备注：并发使用，全部 goroutine 共享全局的只读 vIndex 缓存，每个 goroutine 创建一个独立的 searcher 对象
```
### 缓存整个 `xdb` 数据
可以预先加载整个 ip2region.xdb 到内存，完全基于内存查询，类似于之前的 memory search 查询。
```golang
// 1、从 dbPath 加载整个 xdb 到内存
cBuff, err := LoadContentFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load content from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 cBuff 创建完全基于内存的查询对象。
searcher, err := xdb.NewWithBuffer(cBuff)
if err != nil {
fmt.Printf("failed to create searcher with content: %s\n", err)
return
}
// 备注：并发使用，用整个 xdb 缓存创建的 searcher 对象可以安全用于并发。
```
# 编译测试程序
通过如下方式编译得到 xdb_searcher 可执行程序：
```bash
# 切换到 golang binding 根目录
make
```
# 查询测试
通过 `xdb_searcher search` 命令来测试 ip2region.xdb 的查询：
```
➜ golang git:(v2.0_xdb) ./xdb_searcher search
./xdb_searcher search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 进行查询测试
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_searcher search --db=../../data/ip2region.xdb
ip2region xdb searcher test program, type `quit` to exit
ip2region>> 1.2.3.4
{region:美国|0|华盛顿|0|谷歌, took:101.57µs}
```
输入 ip 地址进行查询即可，输入 quit 退出测试程序。可以设置 `cache-policy` 为 file/vectorIndex/content 来测试不同的查询缓存机制。
# bench 测试
通过 `xdb_searcher bench` 命令来进行自动 bench 测试，一方面确保程序和 `xdb` 文件都没有错误，另一方面通过大量的查询得到平均查询性能：
```bash
➜ golang git:(v2.0_xdb) ./xdb_searcher bench
./xdb_searcher bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过 data/ip2region.xdb 和 data/ip.merge.txt 进行 bench 测试：
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_searcher bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt
Bench finished, {total: 3417955, took: 28.211578339s, cost: 8253 ns/op}
```
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。
*请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件*。
bench 程序会逐行读取 `src` 指定的源IP文件，然后每个 IP 段选取 5 个固定位置的 IP 进行测试，以确保查询的 region 信息和原始的 region 信息是相同。测试途中没有调试信息的输出，有错误会打印错误信息并且终止运行，所以看到 `Bench finished` 就表示 bench 成功了，cost 是表示每次查询操作的平均时间(ns)。
binding/golang/go.mod 0 → 100644
浏览文件 @ 65edd65b
module github.com/lionsoul2014/ip2region/binding/golang
go 1.17
require github.com/mitchellh/go-homedir v1.1.0
binding/golang/go.sum 0 → 100644
浏览文件 @ 65edd65b
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
binding/golang/main.go
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package main package main
import ( import (
"os"
"github.com/lionsoul2014/ip2region/binding/golang/ip2region"
"bufio" "bufio"
"fmt" "fmt"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"github.com/mitchellh/go-homedir"
"log"
"os"
"strings" "strings"
"errors"
"time" "time"
) )
func main() { func printHelp() {
fmt.Printf("ip2region xdb searcher\n")
fmt.Printf("%s [command] [command options]\n", os.Args[0])
fmt.Printf("Command: \n")
fmt.Printf(" search search input test\n")
fmt.Printf(" bench search bench test\n")
}
func testSearch() {
var err error
var dbFile, cachePolicy = "", "vectorIndex"
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default:
fmt.Printf("undefined option `%s`\n", r)
return
}
}
db := os.Args[1] if dbFile == "" {
fmt.Printf("%s search [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
_,err:= os.Stat(db) dbPath, err := homedir.Expand(dbFile)
if os.IsNotExist(err){ if err != nil {
panic("not found db " + db) fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
} }
region, err := ip2region.New(db) // create the searcher with the cache policy setting
defer region.Close() searcher, err := createSearcher(dbPath, cachePolicy)
fmt.Println(`initializing if err != nil {
+-------------------------------------------------------+ fmt.Printf("failed to create searcher: %s\n", err.Error())
| ip2region test script | return
| format 'ip type' | }
| type option 'b-tree','binary','memory' default b-tree | defer func() {
| Type 'quit' to exit program | searcher.Close()
+-------------------------------------------------------+`) fmt.Printf("searcher test program exited, thanks for trying\n")
}()
fmt.Printf(`ip2region xdb searcher test program, cachePolicy: %s
type 'quit' to exit
`, cachePolicy)
reader := bufio.NewReader(os.Stdin) reader := bufio.NewReader(os.Stdin)
for { for {
fmt.Print("ip2reginon >> ") fmt.Print("ip2region>> ")
data, _, _ := reader.ReadLine() str, err := reader.ReadString('\n')
begin:= time.Now() if err != nil {
commands := strings.Fields(string(data)) log.Fatalf("failed to read string: %s", err)
ip := ip2region.IpInfo{} }
len := len(commands)
if len == 0{ line := strings.TrimSpace(strings.TrimSuffix(str, "\n"))
if len(line) == 0 {
continue continue
} }
if commands[0] == "quit"{ if line == "quit" {
break break
} }
if !(len > 1) { tStart := time.Now()
commands = append(commands, "b-tree") region, err := searcher.SearchByStr(line)
if err != nil {
fmt.Printf("\x1b[0;31m{err: %s, ioCount: %d}\x1b[0m\n", err.Error(), searcher.GetIOCount())
} else {
fmt.Printf("\x1b[0;32m{region: %s, ioCount: %d, took: %s}\x1b[0m\n", region, searcher.GetIOCount(), time.Since(tStart))
}
}
}
func testBench() {
var err error
var dbFile, srcFile, cachePolicy = "", "", "vectorIndex"
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
} }
switch commands[1] {
case "b-tree": if strings.Index(r, "--") != 0 {
ip, err = region.BtreeSearch(commands[0]) continue
case "binary": }
ip, err = region.BinarySearch(commands[0])
case "memory": var sIdx = strings.Index(r, "=")
ip, err = region.MemorySearch(commands[0]) if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "src":
srcFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default: default:
err = errors.New("parameter error") fmt.Printf("undefined option `%s`\n", r)
return
} }
}
if dbFile == "" || srcFile == "" {
fmt.Printf("%s bench [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --src string source ip text file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
dbPath, err := homedir.Expand(dbFile)
if err != nil {
fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
}
searcher, err := createSearcher(dbPath, cachePolicy)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer func() {
searcher.Close()
}()
handle, err := os.OpenFile(srcFile, os.O_RDONLY, 0600)
if err != nil {
fmt.Printf("failed to open source text file: %s\n", err)
return
}
var count, tStart, costs = int64(0), time.Now(), int64(0)
var scanner = bufio.NewScanner(handle)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
var l = strings.TrimSpace(strings.TrimSuffix(scanner.Text(), "\n"))
var ps = strings.SplitN(l, "|", 3)
if len(ps) != 3 {
fmt.Printf("invalid ip segment line `%s`\n", l)
return
}
sip, err := xdb.CheckIP(ps[0])
if err != nil {
fmt.Printf("check start ip `%s`: %s\n", ps[0], err)
return
}
eip, err := xdb.CheckIP(ps[1])
if err != nil {
fmt.Printf("check end ip `%s`: %s\n", ps[1], err)
return
}
if sip > eip {
fmt.Printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1])
return
}
mip := xdb.MidIP(sip, eip)
for _, ip := range []uint32{sip, xdb.MidIP(sip, mip), mip, xdb.MidIP(mip, eip), eip} {
sTime := time.Now()
region, err := searcher.Search(ip)
if err != nil {
fmt.Printf("failed to search ip '%s': %s\n", xdb.Long2IP(ip), err)
return
}
costs += time.Since(sTime).Nanoseconds()
// check the region info
if region != ps[2] {
fmt.Printf("failed Search(%s) with (%s != %s)\n", xdb.Long2IP(ip), region, ps[2])
return
}
count++
}
}
cost := time.Since(tStart)
fmt.Printf("Bench finished, {cachePolicy: %s, total: %d, took: %s, cost: %d μs/op}\n",
cachePolicy, count, cost, costs/count/1000)
}
func createSearcher(dbPath string, cachePolicy string) (*xdb.Searcher, error) {
switch cachePolicy {
case "nil", "file":
return xdb.NewWithFileOnly(dbPath)
case "vectorIndex":
vIndex, err := xdb.LoadVectorIndexFromFile(dbPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to load vector index from `%s`: %w", dbPath, err)
}
fmt.Println( fmt.Sprintf("\x1b[0;31m%s\x1b[0m",err.Error())) return xdb.NewWithVectorIndex(dbPath, vIndex)
}else{ case "content":
fmt.Println( fmt.Sprintf("\x1b[0;32m%s %s\x1b[0m",ip.String(),time.Since(begin).String())) cBuff, err := xdb.LoadContentFromFile(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to load content from '%s': %w", dbPath, err)
} }
return xdb.NewWithBuffer(cBuff)
default:
return nil, fmt.Errorf("invalid cache policy `%s`, options: file/vectorIndex/content", cachePolicy)
}
}
func main() {
if len(os.Args) < 2 {
printHelp()
return
}
// set the log flag
log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
switch strings.ToLower(os.Args[1]) {
case "search":
testSearch()
case "bench":
testBench()
default:
printHelp()
} }
} }
binding/golang/xdb/searcher.go 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// ip2region database v2.0 searcher.
// @Note this is a Not thread safe implementation.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
"os"
)
const (
HeaderInfoLength = 256
VectorIndexRows = 256
VectorIndexCols = 256
VectorIndexSize = 8
SegmentIndexBlockSize = 14
)
// --- Index policy define
type IndexPolicy int
const (
VectorIndexPolicy IndexPolicy = 1
BTreeIndexPolicy IndexPolicy = 2
)
func (i IndexPolicy) String() string {
switch i {
case VectorIndexPolicy:
return "VectorIndex"
case BTreeIndexPolicy:
return "BtreeIndex"
default:
return "unknown"
}
}
// --- Header define
type Header struct {
// data []byte
Version uint16
IndexPolicy IndexPolicy
CreatedAt uint32
StartIndexPtr uint32
EndIndexPtr uint32
}
func NewHeader(input []byte) (*Header, error) {
if len(input) < 16 {
return nil, fmt.Errorf("invalid input buffer")
}
return &Header{
Version: binary.LittleEndian.Uint16(input),
IndexPolicy: IndexPolicy(binary.LittleEndian.Uint16(input[2:])),
CreatedAt: binary.LittleEndian.Uint32(input[4:]),
StartIndexPtr: binary.LittleEndian.Uint32(input[8:]),
EndIndexPtr: binary.LittleEndian.Uint32(input[12:]),
}, nil
}
// --- searcher implementation
type Searcher struct {
handle *os.File
// header info
header *Header
ioCount int
// use it only when this feature enabled.
// Preload the vector index will reduce the number of IO operations
// thus speedup the search process
vectorIndex []byte
// content buffer.
// running with the whole xdb file cached
contentBuff []byte
}
func baseNew(dbFile string, vIndex []byte, cBuff []byte) (*Searcher, error) {
var err error
// content buff first
if cBuff != nil {
return &Searcher{
vectorIndex: nil,
contentBuff: cBuff,
}, nil
}
// open the xdb binary file
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
return &Searcher{
handle: handle,
vectorIndex: vIndex,
}, nil
}
func NewWithFileOnly(dbFile string) (*Searcher, error) {
return baseNew(dbFile, nil, nil)
}
func NewWithVectorIndex(dbFile string, vIndex []byte) (*Searcher, error) {
return baseNew(dbFile, vIndex, nil)
}
func NewWithBuffer(cBuff []byte) (*Searcher, error) {
return baseNew("", nil, cBuff)
}
func (s *Searcher) Close() {
if s.handle != nil {
err := s.handle.Close()
if err != nil {
return
}
}
}
// GetIOCount return the global io count for the last search
func (s *Searcher) GetIOCount() int {
return s.ioCount
}
// SearchByStr find the region for the specified ip string
func (s *Searcher) SearchByStr(str string) (string, error) {
ip, err := CheckIP(str)
if err != nil {
return "", err
}
return s.Search(ip)
}
// Search find the region for the specified long ip
func (s *Searcher) Search(ip uint32) (string, error) {
// reset the global ioCount
s.ioCount = 0
// locate the segment index block based on the vector index
var il0 = (ip >> 24) & 0xFF
var il1 = (ip >> 16) & 0xFF
var idx = il0*VectorIndexCols*VectorIndexSize + il1*VectorIndexSize
var sPtr, ePtr = uint32(0), uint32(0)
if s.vectorIndex != nil {
sPtr = binary.LittleEndian.Uint32(s.vectorIndex[idx:])
ePtr = binary.LittleEndian.Uint32(s.vectorIndex[idx+4:])
} else if s.contentBuff != nil {
sPtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx:])
ePtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx+4:])
} else {
// read the vector index block
var buff = make([]byte, VectorIndexSize)
err := s.read(int64(HeaderInfoLength+idx), buff)
if err != nil {
return "", fmt.Errorf("read vector index block at %d: %w", HeaderInfoLength+idx, err)
}
sPtr = binary.LittleEndian.Uint32(buff)
ePtr = binary.LittleEndian.Uint32(buff[4:])
}
// fmt.Printf("sPtr=%d, ePtr=%d", sPtr, ePtr)
// binary search the segment index to get the region
var dataLen, dataPtr = 0, uint32(0)
var buff = make([]byte, SegmentIndexBlockSize)
var l, h = 0, int((ePtr - sPtr) / SegmentIndexBlockSize)
for l <= h {
m := (l + h) >> 1
p := sPtr + uint32(m*SegmentIndexBlockSize)
err := s.read(int64(p), buff)
if err != nil {
return "", fmt.Errorf("read segment index at %d: %w", p, err)
}
// decode the data step by step to reduce the unnecessary operations
sip := binary.LittleEndian.Uint32(buff)
if ip < sip {
h = m - 1
} else {
eip := binary.LittleEndian.Uint32(buff[4:])
if ip > eip {
l = m + 1
} else {
dataLen = int(binary.LittleEndian.Uint16(buff[8:]))
dataPtr = binary.LittleEndian.Uint32(buff[10:])
break
}
}
}
//fmt.Printf("dataLen: %d, dataPtr: %d", dataLen, dataPtr)
if dataLen == 0 {
return "", nil
}
// load and return the region data
var regionBuff = make([]byte, dataLen)
err := s.read(int64(dataPtr), regionBuff)
if err != nil {
return "", fmt.Errorf("read region at %d: %w", dataPtr, err)
}
return string(regionBuff), nil
}
// do the data read operation based on the setting.
// content buffer first or will read from the file.
// this operation will invoke the Seek for file based read.
func (s *Searcher) read(offset int64, buff []byte) error {
if s.contentBuff != nil {
cLen := copy(buff, s.contentBuff[offset:])
if cLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
} else {
_, err := s.handle.Seek(offset, 0)
if err != nil {
return fmt.Errorf("seek to %d: %w", offset, err)
}
s.ioCount++
rLen, err := s.handle.Read(buff)
if err != nil {
return fmt.Errorf("handle read: %w", err)
}
if rLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
}
return nil
}
binding/golang/xdb/util.go 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"fmt"
"os"
"strconv"
"strings"
)
var shiftIndex = []int{24, 16, 8, 0}
func CheckIP(ip string) (uint32, error) {
var ps = strings.Split(ip, ".")
if len(ps) != 4 {
return 0, fmt.Errorf("invalid ip address `%s`", ip)
}
var val = uint32(0)
for i, s := range ps {
d, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("the %dth part `%s` is not an integer", i, s)
}
if d < 0 || d > 255 {
return 0, fmt.Errorf("the %dth part `%s` should be an integer bettween 0 and 255", i, s)
}
val |= uint32(d) << shiftIndex[i]
}
// convert the ip to integer
return val, nil
}
func Long2IP(ip uint32) string {
return fmt.Sprintf("%d.%d.%d.%d", (ip>>24)&0xFF, (ip>>16)&0xFF, (ip>>8)&0xFF, ip&0xFF)
}
func MidIP(sip uint32, eip uint32) uint32 {
return uint32((uint64(sip) + uint64(eip)) >> 1)
}
// LoadHeader load the header info from the specified handle
func LoadHeader(handle *os.File) (*Header, error) {
_, err := handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to the header: %w", err)
}
var buff = make([]byte, HeaderInfoLength)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return NewHeader(buff)
}
// LoadHeaderFromFile load header info from the specified db file path
func LoadHeaderFromFile(dbFile string) (*Header, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
header, err := LoadHeader(handle)
if err != nil {
return nil, err
}
_ = handle.Close()
return header, nil
}
// LoadHeaderFromBuff wrap the header info from the content buffer
func LoadHeaderFromBuff(cBuff []byte) (*Header, error) {
return NewHeader(cBuff[0:256])
}
// LoadVectorIndex util function to load the vector index from the specified file handle
func LoadVectorIndex(handle *os.File) ([]byte, error) {
// load all the vector index block
_, err := handle.Seek(HeaderInfoLength, 0)
if err != nil {
return nil, fmt.Errorf("seek to vector index: %w", err)
}
var buff = make([]byte, VectorIndexRows*VectorIndexCols*VectorIndexSize)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadVectorIndexFromFile load vector index from a specified file path
func LoadVectorIndexFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
vIndex, err := LoadVectorIndex(handle)
if err != nil {
return nil, err
}
_ = handle.Close()
return vIndex, nil
}
// LoadContent load the whole xdb content from the specified file handle
func LoadContent(handle *os.File) ([]byte, error) {
// get file size
fi, err := handle.Stat()
if err != nil {
return nil, fmt.Errorf("stat: %w", err)
}
size := fi.Size()
// seek to the head of the file
_, err = handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to get xdb file length: %w", err)
}
var buff = make([]byte, size)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadContentFromFile load the whole xdb content from the specified db file path
func LoadContentFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
cBuff, err := LoadContent(handle)
if err != nil {
return nil, err
}
_ = handle.Close()
return cBuff, nil
}
binding/golang/xdb/util_test.go 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
"net"
"testing"
"time"
)
func TestCheckIP(t *testing.T) {
var str = "29.34.191.255"
ip, err := CheckIP(str)
if err != nil {
t.Errorf("check ip `%s`: %s\n", str, err)
}
netIP := net.ParseIP(str).To4()
if netIP == nil {
t.Fatalf("parse ip `%s` failed", str)
}
u32 := binary.BigEndian.Uint32(netIP)
fmt.Printf("checkip: %d, parseip: %d, isEqual: %v\n", ip, u32, ip == u32)
}
func TestLong2IP(t *testing.T) {
var str = "29.34.191.255"
netIP := net.ParseIP(str).To4()
if netIP == nil {
t.Fatalf("parse ip `%s` failed", str)
}
u32 := binary.BigEndian.Uint32(netIP)
ipStr := Long2IP(u32)
fmt.Printf("originIP: %s, Long2IP: %s, isEqual: %v\n", str, ipStr, ipStr == str)
}
func TestLoadVectorIndex(t *testing.T) {
vIndex, err := LoadVectorIndexFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load vector index: %s\n", err)
return
}
fmt.Printf("vIndex length: %d\n", len(vIndex))
}
func TestLoadContent(t *testing.T) {
buff, err := LoadContentFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load xdb content: %s\n", err)
return
}
fmt.Printf("buff length: %d\n", len(buff))
}
func TestLoadHeader(t *testing.T) {
header, err := LoadHeaderFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load xdb header info: %s\n", err)
return
}
fmt.Printf("Version : %d\n", header.Version)
fmt.Printf("IndexPolicy : %s\n", header.IndexPolicy.String())
fmt.Printf("CreatedAt : %d(%s)\n", header.CreatedAt, time.Unix(int64(header.CreatedAt), 0).Format(time.RFC3339))
fmt.Printf("StartIndexPtr : %d\n", header.StartIndexPtr)
fmt.Printf("EndIndexPtr : %d\n", header.EndIndexPtr)
}
binding/java/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# ip2region xdb java 查询客户端实现
# 使用方式
### maven 仓库：
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>2.6.5</version>
</dependency>
```
### 完全基于文件的查询
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
// 1、创建 searcher 对象
String dbPath = "ip2region.xdb file path";
Searcher searcher = null;
try {
searcher = Searcher.newWithFileOnly(dbPath);
} catch (IOException e) {
System.out.printf("failed to create searcher with `%s`: %s\n", dbPath, e);
return;
}
// 2、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 3、关闭资源
searcher.close();
// 备注：并发使用，每个线程需要创建一个独立的 searcher 对象单独使用。
}
}
```
### 缓存 `VectorIndex` 索引
我们可以提前从 `xdb` 文件中加载出来 `VectorIndex` 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 中预先加载 VectorIndex 缓存，并且把这个得到的数据作为全局变量，后续反复使用。
byte[] vIndex;
try {
vIndex = Searcher.loadVectorIndexFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load vector index from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithVectorIndex(dbPath, vIndex);
} catch (Exception e) {
System.out.printf("failed to create vectorIndex cached searcher with `%s`: %s\n", dbPath, e);
return;
}
// 3、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 4、关闭资源
searcher.close();
// 备注：每个线程需要单独创建一个独立的 Searcher 对象，但是都共享全局的制度 vIndex 缓存。
}
}
```
### 缓存整个 `xdb` 数据
我们也可以预先加载整个 ip2region.xdb 的数据到内存，然后基于这个数据创建查询对象来实现完全基于文件的查询，类似之前的 memory search。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 加载整个 xdb 到内存。
byte[] cBuff;
try {
cBuff = Searcher.loadContentFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load content from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用上述的 cBuff 创建一个完全基于内存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithBuffer(cBuff);
} catch (Exception e) {
System.out.printf("failed to create content cached searcher: %s\n", e);
return;
}
// 3、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 4、关闭资源 - 该 searcher 对象可以安全用于并发，等整个服务关闭的时候再关闭 searcher
// searcher.close();
// 备注：并发使用，用整个 xdb 数据缓存创建的查询对象可以安全的用于并发，也就是你可以把这个 searcher 对象做成全局对象去跨线程访问。
}
}
```
# 编译测试程序
通过 maven 来编译测试程序。
```bash
# cd 到 java binding 的根目录
cd binding/java/
mvn compile package
```
然后会在当前目录的 target 目录下得到一个 ip2region-{version}.jar 的打包文件。
# 查询测试
可以通过 `java -jar ip2region-{version}.jar search` 命令来测试查询：
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar search
java -jar ip2region-{version}.jar search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 文件进行查询测试：
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar search --db=../../data/ip2region.xdb
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, ioCount: 7, took: 82 μs}
ip2region>>
```
输入 ip 即可进行查询测试，也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的查询效果。
# bench 测试
可以通过 `java -jar ip2region-{version}.jar bench` 命令来进行 bench 测试，一方面确保 `xdb` 文件没有错误，一方面可以评估查询性能：
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar bench
java -jar ip2region-{version}.jar bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 文件进行 bench 测试：
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 8s, cost: 2 μs/op}
```
可以通过分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效果。
@Note: 注意 bench 使用的 src 文件要是生成对应 xdb 文件相同的源文件。
binding/java/pom.xml
浏览文件 @ 65edd65b
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<groupId>org.lionsoul</groupId> <groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId> <artifactId>ip2region</artifactId>
<version>1.7.2</version> <version>2.6.5</version>
<packaging>jar</packaging> <packaging>jar</packaging>
<name>ip2region</name> <name>ip2region</name>
<url>http://gitee.com/lionsoul/ip2region/</url> <url>https://github.com/lionsoul2014/ip2region</url>
<description>Open source internel address locator</description> <description>Open source offline internet address db manager framework and locator</description>
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<scm>
<url>git@gitee.com:lionsoul/ip2region.git</url>
<connection>scm:git:git@gitee.com:lionsoul/ip2region.git</connection>
<developerConnection>scm:git:git@gitee.com:lionsoul/ip2region.git</developerConnection>
</scm>
<developers>
<developer>
<id>lionsoul</id>
<name>chenxin</name>
<email>chenxin619315@gmail.com</email>
</developer>
</developers>
<issueManagement>
<url>http://gitee.com/lionsoul/ip2region/issues</url>
<system>gitee issues</system>
</issueManagement>
<properties> <licenses>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <license>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <name>The Apache Software License, Version 2.0</name>
</properties> <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<dependencies> <scm>
<dependency> <url>git@github.com:lionsoul2014/ip2region.git</url>
<groupId>junit</groupId> <connection>scm:git:git@github.com:lionsoul2014/ip2region.git</connection>
<artifactId>junit</artifactId> <developerConnection>scm:git:git@github.com:lionsoul2014/ip2region.git</developerConnection>
<version>4.13.1</version> </scm>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.1.2</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9</version>
<executions>
<execution>
<id>attach-javadocs</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
<configuration>
<failOnError>false</failOnError>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.lionsoul.ip2region.test.TestSearcher</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles> <developers>
<profile> <developer>
<id>java8-doclint-disabled</id> <id>lionsoul</id>
<activation> <name>chenxin</name>
<jdk>[1.8,)</jdk> <email>chenxin619315@gmail.com</email>
</activation> </developer>
<properties> </developers>
<javadoc.opts>-Xdoclint:none</javadoc.opts>
</properties> <issueManagement>
</profile> <url>https://github.com/lionsoul2014/ip2region/issues</url>
<profile> <system>Github issues</system>
<id>release</id> </issueManagement>
<build>
<plugins> <properties>
<!-- Source --> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<plugin> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<groupId>org.apache.maven.plugins</groupId> </properties>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version> <dependencies>
<executions> <dependency>
<execution> <groupId>junit</groupId>
<phase>package</phase> <artifactId>junit</artifactId>
<goals> <version>4.13.1</version>
<goal>jar-no-fork</goal> <scope>test</scope>
</goals> </dependency>
</execution> </dependencies>
</executions>
</plugin> <build>
<!-- Javadoc --> <plugins>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId> <artifactId>maven-source-plugin</artifactId>
<version>2.9.1</version> <version>2.1.2</version>
<executions> <executions>
<execution> <execution>
<phase>package</phase> <id>attach-sources</id>
<goals> <phase>package</phase>
<goal>jar</goal> <goals>
</goals> <goal>jar</goal>
<configuration> </goals>
<additionalparam>${javadoc.opts}</additionalparam> </execution>
</configuration> </executions>
</execution> </plugin>
</executions>
</plugin> <plugin>
<!-- GPG --> <groupId>org.apache.maven.plugins</groupId>
<plugin> <artifactId>maven-javadoc-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId> <version>2.9</version>
<artifactId>maven-gpg-plugin</artifactId> <executions>
<version>1.5</version> <execution>
<executions> <id>attach-javadocs</id>
<execution> <phase>package</phase>
<phase>verify</phase> <goals>
<goals> <goal>jar</goal>
<goal>sign</goal> </goals>
</goals> <configuration>
</execution> <additionalparam>${javadoc.opts}</additionalparam>
</executions> </configuration>
</plugin> </execution>
</plugins> </executions>
</build> <configuration>
<distributionManagement> <failOnError>false</failOnError>
<snapshotRepository> </configuration>
<id>oss-parent</id> </plugin>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</snapshotRepository> <plugin>
<repository> <groupId>org.apache.maven.plugins</groupId>
<id>oss-parent</id> <artifactId>maven-shade-plugin</artifactId>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url> <version>1.4</version>
</repository> <executions>
</distributionManagement> <execution>
</profile> <phase>package</phase>
</profiles> <goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.lionsoul.ip2region.SearchTest</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>java8-doclint-disabled</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<javadoc.opts>-Xdoclint:none</javadoc.opts>
</properties>
</profile>
<profile>
<id>release</id>
<build>
<plugins>
<!-- Source -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Javadoc -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
</plugin>
<!-- GPG -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<distributionManagement>
<snapshotRepository>
<id>oss</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</snapshotRepository>
<repository>
<id>oss</id>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</distributionManagement>
</profile>
</profiles>
</project> </project>
binding/java/src/main/java/org/lionsoul/ip2region/SearchTest.java 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region;
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearchTest {
public static void printHelp(String[] args) {
System.out.print("ip2region xdb searcher\n");
System.out.print("java -jar ip2region-{version}.jar [command] [command options]\n");
System.out.print("Command: \n");
System.out.print(" search search input test\n");
System.out.print(" bench search bench test\n");
}
public static Searcher createSearcher(String dbPath, String cachePolicy) throws IOException {
if ("file".equals(cachePolicy)) {
return Searcher.newWithFileOnly(dbPath);
} else if ("vectorIndex".equals(cachePolicy)) {
byte[] vIndex = Searcher.loadVectorIndexFromFile(dbPath);
return Searcher.newWithVectorIndex(dbPath, vIndex);
} else if ("content".equals(cachePolicy)) {
byte[] cBuff = Searcher.loadContentFromFile(dbPath);
return Searcher.newWithBuffer(cBuff);
} else {
throw new IOException("invalid cache policy `" + cachePolicy + "`, options: file/vectorIndex/content");
}
}
public static void searchTest(String[] args) throws IOException {
String dbPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
// System.out.printf("key=%s, val=%s\n", key, val);
if ("db".equals(key)) {
dbPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.length() < 1) {
System.out.print("java -jar ip2region-{version}.jar search [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
final BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
System.out.printf("ip2region xdb searcher test program, cachePolicy: %s\ntype 'quit' to exit\n", cachePolicy);
while ( true ) {
System.out.print("ip2region>> ");
String line = reader.readLine().trim();
if ( line.length() < 2 ) {
continue;
}
if ( line.equalsIgnoreCase("quit") ) {
break;
}
try {
double sTime = System.nanoTime();
String region = searcher.search(line);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("{err: %s, ioCount: %d}\n", e, searcher.getIOCount());
}
}
reader.close();
searcher.close();
System.out.println("searcher test program exited, thanks for trying");
}
public static void benchTest(String[] args) throws IOException {
String dbPath = "", srcPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
if ("db".equals(key)) {
dbPath = val;
} else if ("src".equals(key)) {
srcPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.length() < 1 || srcPath.length() < 1) {
System.out.print("java -jar ip2region-{version}.jar bench [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --src string source ip text file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
long count = 0, costs = 0, tStart = System.nanoTime();
String line;
final BufferedReader reader = new BufferedReader(new FileReader(srcPath));
while ((line = reader.readLine()) != null) {
String l = line.trim();
String[] ps = l.split("\\|", 3);
if (ps.length != 3) {
System.out.printf("invalid ip segment `%s`\n", l);
return;
}
long sip;
try {
sip = Searcher.checkIP(ps[0]);
} catch (Exception e) {
System.out.printf("check start ip `%s`: %s\n", ps[0], e);
return;
}
long eip;
try {
eip = Searcher.checkIP(ps[1]);
} catch (Exception e) {
System.out.printf("check end ip `%s`: %s\n", ps[1], e);
return;
}
if (sip > eip) {
System.out.printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1]);
return;
}
long mip = (sip + eip) >> 1;
for (final long ip : new long[]{sip, (sip + mip) >> 1, mip, (mip + eip) >> 1, eip}) {
long sTime = System.nanoTime();
String region = searcher.search(ip);
costs += System.nanoTime() - sTime;
// check the region info
if (!ps[2].equals(region)) {
System.out.printf("failed search(%s) with (%s != %s)\n", Searcher.long2ip(ip), region, ps[2]);
return;
}
count++;
}
}
reader.close();
searcher.close();
long took = System.nanoTime() - tStart;
System.out.printf("Bench finished, {cachePolicy: %s, total: %d, took: %ds, cost: %d μs/op}\n",
cachePolicy, count, TimeUnit.NANOSECONDS.toSeconds(took),
count == 0 ? 0 : TimeUnit.NANOSECONDS.toMicros(costs/count));
}
public static void main(String[] args) {
if (args.length < 1) {
printHelp(args);
return;
}
if ("search".equals(args[0])) {
try {
searchTest(args);
} catch (IOException e) {
System.out.printf("failed running search test: %s\n", e);
}
} else if ("bench".equals(args[0])) {
try {
benchTest(args);
} catch (IOException e) {
System.out.printf("failed running bench test: %s\n", e);
}
} else {
printHelp(args);
}
}
}
binding/java/src/main/java/org/lionsoul/ip2region/UtilTest.java 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region;
import org.lionsoul.ip2region.xdb.Searcher;
public class UtilTest {
public static void testIP2Long() {
String ip = "1.2.3.4";
long ipAddr = 0;
try {
ipAddr = Searcher.checkIP(ip);
} catch (Exception e) {
System.out.printf("failed to check ip: %s\n", e);
return;
}
if (ipAddr != 16909060) {
System.out.print("failed ip2long\n");
return;
}
String ip2 = Searcher.long2ip(ipAddr);
if (!ip.equals(ip2)) {
System.out.print("failed long2ip\n");
return;
}
System.out.printf("passed: ip=%s, ipAddr=%d, ip2=%s\n", ip, ipAddr, ip2);
}
public static void main(String[] args) {
System.out.print("testing IP2Long ... \n");
testIP2Long();
}
}
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Header.java 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region.xdb;
import java.awt.image.SampleModel;
public class Header {
public final int version;
public final int indexPolicy;
public final int createdAt;
public final int startIndexPtr;
public final int endIndexPtr;
public final byte[] buffer;
public Header(byte[] buff) {
assert buff.length >= 16;
version = Searcher.getInt2(buff, 0);
indexPolicy = Searcher.getInt2(buff, 2);
createdAt = Searcher.getInt(buff, 4);
startIndexPtr = Searcher.getInt(buff, 8);
endIndexPtr = Searcher.getInt(buff, 12);
buffer = buff;
}
@Override public String toString() {
return "{" +
"Version: " + version + ',' +
"IndexPolicy: " + indexPolicy + ',' +
"CreatedAt: " + createdAt + ',' +
"StartIndexPtr: " + startIndexPtr + ',' +
"EndIndexPtr: " + endIndexPtr +
'}';
}
}
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Searcher.java 0 → 100644
浏览文件 @ 65edd65b
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// xdb searcher (Not thread safe implementation)
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
import java.io.IOException;
import java.io.RandomAccessFile;
public class Searcher {
// constant defined copied from the xdb maker
public static final int HeaderInfoLength = 256;
public static final int VectorIndexRows = 256;
public static final int VectorIndexCols = 256;
public static final int VectorIndexSize = 8;
public static final int SegmentIndexSize = 14;
// random access file handle for file based search
private final RandomAccessFile handle;
private int ioCount = 0;
// vector index.
// use the byte[] instead of VectorIndex entry array to keep
// the minimal memory allocation.
private final byte[] vectorIndex;
// xdb content buffer, used for in-memory search
private final byte[] contentBuff;
// --- static method to create searchers
public static Searcher newWithFileOnly(String dbPath) throws IOException {
return new Searcher(dbPath, null, null);
}
public static Searcher newWithVectorIndex(String dbPath, byte[] vectorIndex) throws IOException {
return new Searcher(dbPath, vectorIndex, null);
}
public static Searcher newWithBuffer(byte[] cBuff) throws IOException {
return new Searcher(null, null, cBuff);
}
// --- End of creator
public Searcher(String dbFile, byte[] vectorIndex, byte[] cBuff) throws IOException {
if (cBuff != null) {
this.handle = null;
this.vectorIndex = null;
this.contentBuff = cBuff;
} else {
this.handle = new RandomAccessFile(dbFile, "r");
this.vectorIndex = vectorIndex;
this.contentBuff = null;
}
}
public void close() throws IOException {
if (this.handle != null) {
this.handle.close();
}
}
public int getIOCount() {
return ioCount;
}
public String search(String ipStr) throws Exception {
long ip = checkIP(ipStr);
return search(ip);
}
public String search(long ip) throws IOException {
// reset the global counter
this.ioCount = 0;
// locate the segment index block based on the vector index
int sPtr = 0, ePtr = 0;
int il0 = (int) ((ip >> 24) & 0xFF);
int il1 = (int) ((ip >> 16) & 0xFF);
int idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
// System.out.printf("il0: %d, il1: %d, idx: %d\n", il0, il1, idx);
if (vectorIndex != null) {
sPtr = getInt(vectorIndex, idx);
ePtr = getInt(vectorIndex, idx + 4);
} else if (contentBuff != null) {
sPtr = getInt(contentBuff, HeaderInfoLength + idx);
ePtr = getInt(contentBuff, HeaderInfoLength + idx + 4);
} else {
final byte[] buff = new byte[VectorIndexSize];
read(HeaderInfoLength + idx, buff);
sPtr = getInt(buff, 0);
ePtr = getInt(buff, 4);
}
// System.out.printf("sPtr: %d, ePtr: %d\n", sPtr, ePtr);
// binary search the segment index block to get the region info
final byte[] buff = new byte[SegmentIndexSize];
int dataLen = -1, dataPtr = -1;
int l = 0, h = (ePtr - sPtr) / SegmentIndexSize;
while (l <= h) {
int m = (l + h) >> 1;
int p = sPtr + m * SegmentIndexSize;
// read the segment index
read(p, buff);
long sip = getIntLong(buff, 0);
if (ip < sip) {
h = m - 1;
} else {
long eip = getIntLong(buff, 4);
if (ip > eip) {
l = m + 1;
} else {
dataLen = getInt2(buff, 8);
dataPtr = getInt(buff, 10);
break;
}
}
}
// empty match interception
// System.out.printf("dataLen: %d, dataPtr: %d\n", dataLen, dataPtr);
if (dataPtr < 0) {
return null;
}
// load and return the region data
final byte[] regionBuff = new byte[dataLen];
read(dataPtr, regionBuff);
return new String(regionBuff, "utf-8");
}
protected void read(int offset, byte[] buffer) throws IOException {
// check the in-memory buffer first
if (contentBuff != null) {
// @TODO: reduce data copying, directly decode the data ?
System.arraycopy(contentBuff, offset, buffer, 0, buffer.length);
return;
}
// read from the file handle
assert handle != null;
handle.seek(offset);
this.ioCount++;
int rLen = handle.read(buffer);
if (rLen != buffer.length) {
throw new IOException("incomplete read: read bytes should be " + buffer.length);
}
}
// --- static cache util function
public static Header loadHeader(RandomAccessFile handle) throws IOException {
handle.seek(0);
final byte[] buff = new byte[HeaderInfoLength];
handle.read(buff);
return new Header(buff);
}
public static Header loadHeaderFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final Header header = loadHeader(handle);
handle.close();
return header;
}
public static byte[] loadVectorIndex(RandomAccessFile handle) throws IOException {
handle.seek(HeaderInfoLength);
int len = VectorIndexRows * VectorIndexCols * SegmentIndexSize;
final byte[] buff = new byte[len];
int rLen = handle.read(buff);
if (rLen != len) {
throw new IOException("incomplete read: read bytes should be " + len);
}
return buff;
}
public static byte[] loadVectorIndexFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final byte[] vIndex = loadVectorIndex(handle);
handle.close();
return vIndex;
}
public static byte[] loadContent(RandomAccessFile handle) throws IOException {
handle.seek(0);
final byte[] buff = new byte[(int) handle.length()];
int rLen = handle.read(buff);
if (rLen != buff.length) {
throw new IOException("incomplete read: read bytes should be " + buff.length);
}
return buff;
}
public static byte[] loadContentFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final byte[] content = loadContent(handle);
handle.close();
return content;
}
// --- End cache load util function
// --- static util method
/* get an int from a byte array start from the specified offset */
public static long getIntLong(byte[] b, int offset) {
return (
((b[offset++] & 0x000000FFL)) |
((b[offset++] << 8) & 0x0000FF00L) |
((b[offset++] << 16) & 0x00FF0000L) |
((b[offset ] << 24) & 0xFF000000L)
);
}
public static int getInt(byte[] b, int offset) {
return (
((b[offset++] & 0x000000FF)) |
((b[offset++] << 8) & 0x0000FF00) |
((b[offset++] << 16) & 0x00FF0000) |
((b[offset ] << 24) & 0xFF000000)
);
}
public static int getInt2(byte[] b, int offset) {
return (
(b[offset++] & 0x000000FF) |
(b[offset ] & 0x0000FF00)
);
}
/* long int to ip string */
public static String long2ip( long ip )
{
return String.valueOf((ip >> 24) & 0xFF) + '.' +
((ip >> 16) & 0xFF) + '.' + ((ip >> 8) & 0xFF) + '.' + ((ip) & 0xFF);
}
public static final byte[] shiftIndex = {24, 16, 8, 0};
/* check the specified ip address */
public static long checkIP(String ip) throws Exception {
String[] ps = ip.split("\\.");
if (ps.length != 4) {
throw new Exception("invalid ip address `" + ip + "`");
}
long ipDst = 0;
for (int i = 0; i < ps.length; i++) {
int val = Integer.parseInt(ps[i]);
if (val > 255) {
throw new Exception("ip part `"+ps[i]+"` should be less then 256");
}
ipDst |= ((long) val << shiftIndex[i]);
}
return ipDst & 0xFFFFFFFFL;
}
}
\ No newline at end of file
binding/lua/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
# ip2region xdb lua 查询客户端实现
#### 备注：请优先使用 lua_c 扩展 xdb 查询客户端，性能比纯 lua 实现的要快很多！！！
# 使用方式
### 完全基于文件的查询
```lua
local xdb = require("xdb_searcher")
-- 1、从 db_path 创建基于文件的 xdb 查询对象
local db_path = "ip2region.xdb file path"
local searcher, err = xdb.new_with_file_only(db_path)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
-- 2、调用查询 API 进行查询
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
### 缓存 `VectorIndex` 索引
如果你的 `lua` 母环境支持，可以预先加载 vectorIndex 缓存，然后做成全局变量，每次创建 Searcher 的时候使用全局的 vectorIndex，可以减少一次固定的 IO 操作从而加速查询，减少 io 压力。
```lua
local xdb = require("xdb_searcher")
local db_path = "ip2region.xdb file path"
-- 1、从指定的 db_path 加载 VectorIndex 缓存，把下述的 v_index 对象做成全局变量。
-- vectorIndex 加载一次即可，建议在服务启动的时候加载为全局对象。
v_index, err = xdb.load_vector_index(db_path)
if err ~= nil then
print(string.format("failed to load vector index from '%s'", db_path))
return
end
-- 2、使用全局的 v_index 创建带 VectorIndex 缓存的查询对象。
searcher, err = xdb.new_with_vector_index(db_path, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
-- 3、调用查询 API
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象，但是共享全局的 v_index 对象
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
### 缓存整个 `xdb` 数据
如果你的 `lua` 母环境支持，可以预先加载整个 xdb 的数据到内存，这样可以实现完全基于内存的查询，类似之前的 memory search 查询。
```lua
local xdb = require("xdb_searcher")
local db_path = "ip2region.xdb file path"
-- 1、从指定的 db_path 加载整个 xdb 到内存。
-- xdb内容加载一次即可，建议在服务启动的时候加载为全局对象。
content = xdb.load_content(db_path)
if content == nil then
print(string.format("failed to load xdb content from '%s'", db_path))
return
end
-- 2、使用全局的 content 创建带完全基于内存的查询对象。
searcher, err = xdb.new_with_buffer(content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
-- 3、调用查询 API
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，用 xdb 整个缓存创建的查询对象可以安全的用于并发。
-- 建议在服务启动的时候创建好全局的 searcher 对象，然后全局并发使用。
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
# 查询测试
通过 `search_test.lua` 脚本来进行查询测试：
```bash
➜ lua git:(lua_binding) ✗ lua search_test.lua
lua search_test.lua [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 进行查询测试：
```bash
➜ lua git:(lua_binding) ✗ lua search_test.lua --db=../../data/ip2region.xdb --cache-policy=vectorIndex
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, io_count: 7, took: 0μs}
ip2region>>
```
输入 ip 即可进行查询测试。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效率。
# bench 测试
通过 `bench_test.lua` 脚本来进行自动 bench 测试，一方面确保 `xdb` 文件没有错误，另一方面通过大量的查询测试平均查询性能：
```bash
➜ lua git:(lua_binding) ✗ lua bench_test.lua
lua bench_test.lua [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 来进行 bench 测试：
```bash
➜ lua git:(lua_binding) ✗ lua bench_test.lua --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt --cache-policy=vectorIndex
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 29.000 s, cost: 7.899 μs/op}
```
可以通过设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 三种不同的缓存实现的的性能。
@Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件。
binding/lua/bench_test.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua/search_test.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua/util_test.lua 0 → 100644
浏览文件 @ 65edd65b
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/07/05
-- set the package path
package.path = "./?.lua"
package.cpath = "./?.so"
local xdb = require("xdb_searcher")
---- ip checking testing
print("--- testing check_ip and long2ip ... ")
local ip_list = {
"1.2.3.4", "192.168.2.3", "120.24.78.129", "255.255.255.0",
"256.7.12.9", "12.56.78.320", "32.12.45.192", "222.221.220.219",
"192.168.1.101 ", "132.96.12.98a", "x23.12.2.12"
}
local s_time = xdb.now()
for _, ip_src in ipairs(ip_list) do
ip, err = xdb.check_ip(ip_src)
if err ~= nil then
print(string.format("invalid ip address `%s`: %s", ip_src, err))
else
ip_dst = xdb.long2ip(ip)
io.write(string.format("long(%-15s)=%10d, long2ip(%-10d)=%-15s", ip_src, ip, ip, ip_dst))
if ip_src ~= ip_dst then
print(" --[Failed]")
else
print(" --[Ok]")
end
end
end
---- buffer loading test
print("\n--- testing load header ... ")
header, err = xdb.load_header("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load header: ", err)
else
print("xdb header buffer loaded")
local tpl = [[
header: {
version: %d
index_policy: %d
created_at: %d
start_index_ptr: %d
end_index_ptr: %d
}]]
print(string.format(tpl,
header["version"], header["index_policy"],
header["created_at"], header["start_index_ptr"], header["end_index_ptr"])
)
end
print("\n--- testing load vector index ... ")
v_index, err = xdb.load_vector_index("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load vector index: ", err)
else
print("xdb vector index buffer loaded")
end
print("\n--- testing load content buffer ... ")
c_buffer, err = xdb.load_content("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load content: ", err)
else
print("xdb content buffer loaded")
end
print("\n--- testing search ... ")
local ip_str = "1.2.3.4"
searcher, err = xdb.new_with_file_only("../../data/ip2region.xdb")
local t_start = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("search(%s) failed: %s", ip_str, err))
else
local c_time = xdb.now() - t_start
print(string.format("search(%s): {region=%s, io_count: %d, took: %dμs, err=%s}",
ip_str, region, searcher:get_io_count(), c_time, err))
print(string.format("searcher.tostring=%s", searcher))
end
searcher:close()
print("")
print(string.format("all tests done, elapsed %d μs", xdb.now() - s_time))
\ No newline at end of file
binding/lua/xdb_searcher.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/Makefile
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/bench_test.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/search_test.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/util_test.lua 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/lua_c/xdb_searcher.c 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/.eslintrc.js 0 → 100644
浏览文件 @ 65edd65b
module.exports = {
env: {
browser: true,
commonjs: true,
es2021: true
},
parserOptions: {
ecmaVersion: 'latest'
},
extends: [
'standard'
],
globals: {
describe: true,
it: true
}
}
binding/nodejs/.mocharc.json 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/.nycrc.json 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/index.js 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/package.json
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/tests/bench.app.js 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/tests/benchmark.js 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/tests/function.test.js 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/nodejs/tests/test.app.js 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php/XdbSearcher.class.php 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php/bench_test.php 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php/search_test.php 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php/util_test.php 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php5_ext/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/php7_ext/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/python/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/python/bench_test.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/python/iptest.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/python/search_test.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/python/xdbSearcher.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
binding/rust/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
data/ip2region.xdb 0 → 100644
浏览文件 @ 65edd65b
文件已添加
maker/c/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/Makefile 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/go.mod 0 → 100644
浏览文件 @ 65edd65b
module github.com/lionsoul2014/ip2region/maker/golang
go 1.17
maker/golang/main.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/make.bat 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/index.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/maker.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/searcher.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/segment.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/util.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/golang/xdb/util_test.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/java/ReadMe.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/java/pom.xml 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/README.md 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/main.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/xdb/__init__.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/xdb/index.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/xdb/maker.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/xdb/segment.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
maker/python/xdb/util.py 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/binding/c/Makefile 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/binding/golang/main.go 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/binding/java/pom.xml 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/binding/lua_c/Makefile 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/binding/nodejs/package.json 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/data/global_region.csv 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。
v1.0/data/ip.merge.txt 0 → 100644
浏览文件 @ 65edd65b
此差异已折叠。