首先安装mongod3.6
vi /etc/yum.repos.d/mongodb-org-3.6.repo
写入以下内容：
[mongodb-org-3.6] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
安装 MongoDB 软件包
yum install mongodb-org -y
为防止意外升级，请固定包装。要固定包，请在您的/etc/yum.conf文件中添加以下exclude指令：
exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools
手动安装包:
wget https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.6/x86_64/RPMS/mongodb-org-3.6.23-1.el7.x86_64.rpm wget https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.6/x86_64/RPMS/mongodb-org-mongos-3.6.23-1.el7.x86_64.rpm wget https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.6/x86_64/RPMS/mongodb-org-server-3.6.23-1.el7.x86_64.rpm wget https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.6/x86_64/RPMS/mongodb-org-shell-3.6.23-1.el7.x86_64.rpm wget https://repo.mongodb.org/yum/redhat/7/mongodb-org/3.6/x86_64/RPMS/mongodb-org-tools-3.6.23-1.el7.x86_64.rpm yum localinstall *.rpm
启动数据库:
systemctl start mongod
查看版本:
mongod --version
安装python3.x
编译环境
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make -y
去官网下载编译安装包或者直接执行以下命令下载
wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tar.xz
解压
tar -xvJf Python-3.6.8.tar.xz
切换进入
cd Python-3.6.8
编译安装
./configure prefix=/usr/local/python3 make && make install
安装完毕，/usr/local/目录下就会有python3了
因此我们可以添加软链到执行目录下/usr/bin
ln -s /usr/local/python3/bin/python3 /usr/bin/python3 ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
安装Docker
#CentOS 7、Debian、Ubuntu系统
curl -sSL https://get.docker.com/ | sh systemctl start docker systemctl enable docker
拉取源码：
git clone https://github.com/ludashi2020/btSearch
在/etc/sysctl.conf文件最后添加一行
vm.max_map_count=262144
执行
/sbin/sysctl -p
立即生效
Elasticsearch dockerp配置
docker run --restart=always -p 9200:9200 -p 9300:9300 --name=tmp docker.elastic.co/elasticsearch/elasticsearch:5.6.0
安装analysis-ik分词器
docker ps docker exec -it $dockid /bin/bash ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.0/elasticsearch-analysis-ik-5.6.0.zip
或者执行
./bin/elasticsearch-plugin install https://get.infini.cloud/elasticsearch/analysis-ik/5.6.0
配置elasticsearch的数据存放目录
mkdir -p /data/docker docker cp $dockid:/usr/share/elasticsearch/ /data/docker chmod 777 -R /data/
停止docker服务
docker stop $dockid docker rm $dockid
检查 elasticsearch.yml 配置文件：
action.auto_create_index: true
运行Elasticsearch docker服务
docker run --restart=always -p 9200:9200 -p 9300:9300 --name=es \ -e ES_JAVA_OPTS="-Xms4g -Xmx4g" \ -v /data/docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /data/docker/elasticsearch/data:/usr/share/elasticsearch/data \ -v /data/docker/elasticsearch/logs:/usr/share/elasticsearch/logs \ -v /data/docker/elasticsearch/config:/usr/share/elasticsearch/config \ -v /data/docker/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -v /etc/localtime:/etc/localtime \ -v /etc/timezone:/etc/timezone \ docker.elastic.co/elasticsearch/elasticsearch:5.6.0
如果需要绑定127.0.0.1就执行下面的命令：
docker run --restart=always -p 127.0.0.1:9200:9200 -p 127.0.0.1:9300:9300 --name=es \ -e ES_JAVA_OPTS="-Xms4g -Xmx4g" \ -v /data/docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /data/docker/elasticsearch/data:/usr/share/elasticsearch/data \ -v /data/docker/elasticsearch/logs:/usr/share/elasticsearch/logs \ -v /data/docker/elasticsearch/config:/usr/share/elasticsearch/config \ -v /data/docker/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -v /etc/localtime:/etc/localtime \ -v /etc/timezone:/etc/timezone \ docker.elastic.co/elasticsearch/elasticsearch:5.6.0
配置正确的话执行下面命令可以看到当前Elasticsearch的运行状态
curl --user elastic:changeme -XGET 'http://127.0.0.1:9200/_cat/health'
授权检查
curl --user elastic:changeme -XGET 'http://127.0.0.1:9200/_xpack/license'
配置Elasticsearch默认分词器
curl --user elastic:changeme -XPUT http://127.0.0.1:9200/bavbt -H 'Content-Type: application/json'
curl --user elastic:changeme -XPOST '127.0.0.1:9200/bavbt/_close'
curl --user elastic:changeme -X PUT "127.0.0.1:9200/bavbt/_settings?pretty" \
-H 'Content-Type: application/json' \
-d '{
"index": {
"analysis": {
"analyzer": {
"default": {
"type": "ik_max_word"
}
},
"search_analyzer": {
"default": {
"type": "ik_max_word"
}
}
}
}
}'
curl --user elastic:changeme -XPUT http://127.0.0.1:9200/bavbt/_settings?preserve_existing=true -H 'Content-Type: application/json' -d '{
"max_result_window" : "2000000000"
}'
curl --user elastic:changeme -XPOST '127.0.0.1:9200/bavbt/_open'
# 使用默认密码更改为新密码
curl -X PUT -u elastic:changeme 'http://127.0.0.1:9200/_xpack/security/user/elastic/_password' \
-H "Content-Type: application/json" \
-d '{
"password": "Bt8837609"
}'
设置Mongodb配置副本集
systemctl stop mongod vi /etc/mongod.conf
添加
replication: #oplogSizeMB: 97280 replSetName: rs1
启动数据库
systemctl start mongod
设置root密码
mongod --dbpath /var/lib/mongo --logpath /var/log/mongodb/mongod.log --fork
mongo
use admin
db.createUser({user:"root",pwd:"root",roles:[{role:"root",db:"admin"}]})
db.auth("root","root")
初始化副本集
rs.initiate()
显示如下信息，表示正常
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "127.0.0.1:27017",
"ok" : 1,
"operationTime" : Timestamp(1645696651, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1645696651, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
运行
首先需要配置您的conf文件
[mongodb] musername= mpassword= database=bavbt collection=torrent addr=127.0.0.1:27017 [elasticsearch] url=http://127.0.0.1:9200/bavbt/torrent/ eusername=elastic epassword=changeme [webServer] webServerAddr=127.0.0.1:7878
安装mongo-connector，这个东西的主要作用是把mongodb的数据同步到es内：
pip3 install mongo-connector pip3 install elastic2-doc-manager[elastic5] pip3 install mongo-connector[elastic5]
装好之后降级pymongo，否则后续报错运行不了：
pip3 install pymongo==3.12.3
建立短链接
ln -s /usr/local/python3/bin/mongo-connector /usr/bin/mongo-connector
开启同步命令：
根据 Elasticsearch 版本选择对应命令：
情况 1：Elasticsearch ≤ 6.x
mongo-connector -m mongodb://127.0.0.1:27017/?unicode_decode_error_handler=ignore -t http://elastic:[email protected]:9200/bavbt/torrent -d elastic2_doc_manager -n bavbt.torrent -i name,length,hot,create_time,category,infohash
情况 2：Elasticsearch ≥ 7.x
mongo-connector -m mongodb://127.0.0.1:27017/?unicode_decode_error_handler=ignore -t http://elastic:[email protected]:9200/torrent -d elastic2_doc_manager -n bavbt.torrent -i name,length,hot,create_time,category,infohash
ES5.6.0的正确同步命令：
mongo-connector -m mongodb://127.0.0.1:27017/?unicode_decode_error_handler=ignore -t http://elastic:[email protected]:9200 -d elastic2_doc_manager -n bavbt.torrent -i name,length,hot,create_time,category,infohash
验证步骤
1检查索引是否存在：
curl -u elastic:Bt8837609 http://127.0.0.1:9200/_cat/indices?v
2查看索引映射（确认字段同步）：
curl -u elastic:Bt8837609 http://127.0.0.1:9200/bavbt/_mapping?pretty
3监控同步日志：
tail -f /path/to/mongo-connector.log
数据清空
docker stop es sudo rm -rf /data/docker/elasticsearch/data/* docker start es
mongodb数据查询命令：
mongo use bavbt db.torrent.count()