0%

solr5.3.1集群安装

solr5.3.1 集群安装

资料来源https://wiki.apache.org/solr/
下载地址https://archive.apache.org/dist/lucene/solr/5.3.1/
安装 jdk

使用 solr 脚本启动集群

  1. 解压

    1
    2
    3
    4
    tar -zxvf solr-5.3.1.tgz
    mkdir -p /opt/usr/solr
    mv solr-5.3.1 /opt/usr/solr/
    cd /opt/usr/solr/solr-5.3.1
  2. 复制 solr.xml 到 9010 9011 9012 9013 目录下

    创建目录 9010 9011 9012 9013
    mkdir 9010 9011 9012 9013

    1
    2
    3
    4
    cp server/solr/solr.xml 9010/
    cp server/solr/solr.xml 9011/
    cp server/solr/solr.xml 9012/
    cp server/solr/solr.xml 9013/
  3. 启动 solr

    1
    2
    3
    4
    bin/solr -c -p 9010 -s 9010 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
    bin/solr -c -p 9011 -s 9011 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
    bin/solr -c -p 9012 -s 9012 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
    bin/solr -c -p 9013 -s 9013 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
  4. 重启 solr

    1
    2
    3
    4
    5
    6
    7
    bin/solr restart -c -p 9010 -s 9010 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184

    bin/solr restart -c -p 9011 -s 9011 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184

    bin/solr restart -c -p 9012 -s 9012 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184

    bin/solr restart -c -p 9013 -s 9013 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
  5. 停止 solr

    1
    2
    3
    4
    5
    6
    bin/solr stop -p 9010
    bin/solr stop -p 9011
    bin/solr stop -p 9012
    bin/solr stop -p 9013
    或者
    bin/solr stop -all 停止所有
  6. 查看 solr 运行状态

    1
    bin/solr status

    https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s1.png

上传配置文件到 zookeeper

https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files

1
2
mkdir production_conf
cp -rf server/solr/configsets/basic_configs/conf/* production_conf/

上传整个目录

1
server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2182 -confname production_conf -confdir production_conf/

–配置文件在/configs 目录下

上传单个文件

1
server/scripts/cloud-scripts/zkcli.sh -cmd putfile /configs/production_conf/schema.xml production_conf/schema.xml -zkhost 127.0.0.1:2182

https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s2.png

使用配置文件创建 collection

相关 api
https://cwiki.apache.org/confluence/display/solr/Collections+API

创建 collection

1
http://192.168.88.21:9010/solr/admin/collections?action=CREATE&name=production&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=production_conf

https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s3.png

当前 collection 使用的 config

https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s4.png

Reload Colletion(加载配置文件,不需要重启 solr)

/admin/collections?action=RELOAD&name=production

查看 Cluster Status

/admin/collections?action=CLUSTERSTATUS

其它相关操作

添加删除 shard replica 等

https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api18

配置 IK 分词器

复制 IK 分词器到 server/solr-webapp/webapp/WEB-INF/lib
ik 分词器是自己修改编译的

git 地址
https://git.oschina.net/duanzhaoqian/IK-Analyzer-2012FF/tree/master
下载 dist 目录下的 IKAnalyzer-5.3.1.jar

1
cp ~/soft/IKAnalyzer-5.3.1.jar server/solr-webapp/webapp/WEB-INF/lib

修改 schema.xml

1
vim production_conf/schema.xml
1
2
3
4
<fieldType name="text_ik" class="solr.TextField">
<analyzer type="query" class="org.wltea.analyzer.lucene.IKAnalyzer" useSmart="false"/>
<analyzer type="index" class="org.wltea.analyzer.lucene.IKAnalyzer" useSmart="true"/>
</fieldType>

https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s5.png
上传修改完成的 schema.xml

1
server/scripts/cloud-scripts/zkcli.sh -cmd putfile /configs/production_conf/schema.xml production_conf/schema.xml -zkhost 127.0.0.1:2182

Reload Collection

/admin/collections?action=RELOAD&name=production

问题

https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s6.png

找到这三个文件上传到 zookeeper 就好

1
cp example/example-DIH/solr/solr/conf/admin-extra.* production_conf/

上传到 zookeeper

1
server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2182 -confname production_conf -confdir production_conf/

重启 solr 集群

1
2
3
4
bin/solr restart -c -p 9010 -s 9010 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
bin/solr restart -c -p 9011 -s 9011 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
bin/solr restart -c -p 9012 -s 9012 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184
bin/solr restart -c -p 9013 -s 9013 -z 127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184

solrCloud 运行在 HDFS

资料https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS

  1. 添加启动参数(必须)

    1
    -Dsolr.lock.type=hdfs

    https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s7.png
    或者设置配置文件也行

    https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s8.png

  2. 设置 hadoop 配置文件路径(HA 模式必须配置)

    1
    <str name="solr.hdfs.confdir">/opt/solr/etc/hadoop</str>
  3. 修改 DirectoryFactory

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
        	<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">

    <str name="solr.hdfs.home">hdfs://192.168.1.201:8020/solr</str>
    <bool name="solr.hdfs.blockcache.enabled">true</bool>
    <int name="solr.hdfs.blockcache.slab.count">1</int>
    <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
    <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
    <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
    <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
    <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
    <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
    <str name="solr.hdfs.confdir">/opt/solr/etc/hadoop</str>
    </directoryFactory>

    https://raw.githubusercontent.com/duanzhaoqian/pic/master/solr/s9.png

使用 tomcat 启动 solr 集群

solr 服务安装脚本

1
install_solr_service.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [[ $EUID -ne 0 ]]; then
echo -e "\nERROR: This script must be run as root\n" 1>&2
exit 1
fi

print_usage() {
ERROR_MSG="$1"

if [ "$ERROR_MSG" != "" ]; then
echo -e "\nERROR: $ERROR_MSG\n" 1>&2
fi

echo ""
echo "Usage: install_solr_service.sh path_to_solr_distribution_archive OPTIONS"
echo ""
echo " The first argument to the script must be a path to a Solr distribution archive, such as solr-5.0.0.tgz"
echo " (only .tgz or .zip are supported formats for the archive)"
echo ""
echo " Supported OPTIONS include:"
echo ""
echo " -d Directory for live / writable Solr files, such as logs, pid files, and index data; defaults to /var/solr"
echo ""
echo " -i Directory to extract the Solr installation archive; defaults to /opt/"
echo " The specified path must exist prior to using this script."
echo ""
echo " -p Port Solr should bind to; default is 8983"
echo ""
echo " -s Service name; defaults to solr"
echo ""
echo " -u User to own the Solr files and run the Solr process as; defaults to solr"
echo " This script will create the specified user account if it does not exist."
echo ""
echo " NOTE: Must be run as the root user"
echo ""
} # end print_usage

if [ -f "/proc/version" ]; then
proc_version=`cat /proc/version`
else
proc_version=`uname -a`
fi

if [[ $proc_version == *"Debian"* ]]; then
distro=Debian
elif [[ $proc_version == *"Red Hat"* ]]; then
distro=RedHat
elif [[ $proc_version == *"Ubuntu"* ]]; then
distro=Ubuntu
elif [[ $proc_version == *"SUSE"* ]]; then
distro=SUSE
else
echo -e "\nERROR: Your Linux distribution ($proc_version) not supported by this script!\nYou'll need to setup Solr as a service manually using the documentation provided in the Solr Reference Guide.\n" 1>&2
exit 1
fi

if [ -z "$1" ]; then
print_usage "Must specify the path to the Solr installation archive, such as solr-5.0.0.tgz"
exit 1
fi

SOLR_ARCHIVE=$1
if [ ! -f "$SOLR_ARCHIVE" ]; then
print_usage "Specified Solr installation archive $SOLR_ARCHIVE not found!"
exit 1
fi

# strip off path info
SOLR_INSTALL_FILE=${SOLR_ARCHIVE##*/}
is_tar=true
if [ ${SOLR_INSTALL_FILE: -4} == ".tgz" ]; then
SOLR_DIR=${SOLR_INSTALL_FILE%.tgz}
elif [ ${SOLR_INSTALL_FILE: -4} == ".zip" ]; then
SOLR_DIR=${SOLR_INSTALL_FILE%.zip}
is_tar=false
else
print_usage "Solr installation archive $SOLR_ARCHIVE is invalid, expected a .tgz or .zip file!"
exit 1
fi

if [ $# -gt 1 ]; then
shift
while true; do
case $1 in
-i)
if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
print_usage "Directory path is required when using the $1 option!"
exit 1
fi
SOLR_EXTRACT_DIR=$2
shift 2
;;
-d)
if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
print_usage "Directory path is required when using the $1 option!"
exit 1
fi
SOLR_VAR_DIR="$2"
shift 2
;;
-u)
if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
print_usage "Username is required when using the $1 option!"
exit 1
fi
SOLR_USER="$2"
shift 2
;;
-s)
if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
print_usage "Service name is required when using the $1 option!"
exit 1
fi
SOLR_SERVICE="$2"
shift 2
;;
-p)
if [[ -z "$2" || "${2:0:1}" == "-" ]]; then
print_usage "Port is required when using the $1 option!"
exit 1
fi
SOLR_PORT="$2"
shift 2
;;
-help|-usage)
print_usage ""
exit 0
;;
--)
shift
break
;;
*)
if [ "$1" != "" ]; then
print_usage "Unrecognized or misplaced argument: $1!"
exit 1
else
break # out-of-args, stop looping
fi
;;
esac
done
fi

if [ -z "$SOLR_EXTRACT_DIR" ]; then
SOLR_EXTRACT_DIR=/opt
fi

if [ ! -d "$SOLR_EXTRACT_DIR" ]; then
print_usage "Installation directory $SOLR_EXTRACT_DIR not found! Please create it before running this script."
exit 1
fi

if [ -z "$SOLR_SERVICE" ]; then
SOLR_SERVICE=solr
fi

if [ -z "$SOLR_VAR_DIR" ]; then
SOLR_VAR_DIR=/var/$SOLR_SERVICE
fi

if [ -z "$SOLR_USER" ]; then
SOLR_USER=solr
fi

if [ -z "$SOLR_PORT" ]; then
SOLR_PORT=8983
fi

if [ -f "/etc/init.d/$SOLR_SERVICE" ]; then
echo -e "\nERROR: /etc/init.d/$SOLR_SERVICE already exists! Perhaps Solr is already setup as a service on this host?\n" 1>&2
exit 1
fi

if [ -e "$SOLR_EXTRACT_DIR/$SOLR_SERVICE" ]; then
print_usage "$SOLR_EXTRACT_DIR/$SOLR_SERVICE already exists! Please move this directory / link or choose a different service name using the -s option."
exit 1
fi

solr_uid=`id -u $SOLR_USER`
if [ $? -ne 0 ]; then
echo "Creating new user: $SOLR_USER"
if [ "$distro" == "RedHat" ]; then
adduser $SOLR_USER
elif [ "$distro" == "SUSE" ]; then
useradd -m $SOLR_USER
else
adduser --system --shell /bin/bash --group --disabled-password --home /home/$SOLR_USER $SOLR_USER
fi
fi

SOLR_INSTALL_DIR=$SOLR_EXTRACT_DIR/$SOLR_DIR
if [ ! -d "$SOLR_INSTALL_DIR" ]; then

echo "Extracting $SOLR_ARCHIVE to $SOLR_EXTRACT_DIR"

if $is_tar ; then
tar zxf $SOLR_ARCHIVE -C $SOLR_EXTRACT_DIR
else
unzip -q $SOLR_ARCHIVE -d $SOLR_EXTRACT_DIR
fi

if [ ! -d "$SOLR_INSTALL_DIR" ]; then
echo -e "\nERROR: Expected directory $SOLR_INSTALL_DIR not found after extracting $SOLR_ARCHIVE ... script fails.\n" 1>&2
exit 1
fi

chown -R $SOLR_USER: $SOLR_INSTALL_DIR
else
echo -e "\nWARNING: $SOLR_INSTALL_DIR already exists! Skipping extract ...\n"
fi

# create a symlink for easier scripting
ln -s $SOLR_INSTALL_DIR $SOLR_EXTRACT_DIR/$SOLR_SERVICE
chown -h $SOLR_USER: $SOLR_EXTRACT_DIR/$SOLR_SERVICE

mkdir -p $SOLR_VAR_DIR/data
mkdir -p $SOLR_VAR_DIR/logs
cp $SOLR_INSTALL_DIR/server/solr/solr.xml $SOLR_VAR_DIR/data/
cp $SOLR_INSTALL_DIR/bin/solr.in.sh $SOLR_VAR_DIR/
cp $SOLR_INSTALL_DIR/server/resources/log4j.properties $SOLR_VAR_DIR/log4j.properties
sed_expr="s#solr.log=.*#solr.log=\${solr.solr.home}/../logs#"
sed -i -e "$sed_expr" $SOLR_VAR_DIR/log4j.properties
chown -R $SOLR_USER: $SOLR_VAR_DIR

echo "SOLR_PID_DIR=$SOLR_VAR_DIR
SOLR_HOME=$SOLR_VAR_DIR/data
LOG4J_PROPS=$SOLR_VAR_DIR/log4j.properties
SOLR_LOGS_DIR=$SOLR_VAR_DIR/logs
SOLR_PORT=$SOLR_PORT
" >> $SOLR_VAR_DIR/solr.in.sh

echo "Creating /etc/init.d/$SOLR_SERVICE script ..."
cp $SOLR_INSTALL_DIR/bin/init.d/solr /etc/init.d/$SOLR_SERVICE
chmod 744 /etc/init.d/$SOLR_SERVICE
chown root:root /etc/init.d/$SOLR_SERVICE

# do some basic variable substitution on the init.d script
sed_expr1="s#SOLR_INSTALL_DIR=.*#SOLR_INSTALL_DIR=$SOLR_EXTRACT_DIR/$SOLR_SERVICE#"
sed_expr2="s#SOLR_ENV=.*#SOLR_ENV=$SOLR_VAR_DIR/solr.in.sh#"
sed_expr3="s#RUNAS=.*#RUNAS=$SOLR_USER#"
sed_expr4="s#Provides:.*#Provides: $SOLR_SERVICE#"
sed -i -e "$sed_expr1" -e "$sed_expr2" -e "$sed_expr3" -e "$sed_expr4" /etc/init.d/$SOLR_SERVICE

if [[ "$distro" == "RedHat" || "$distro" == "SUSE" ]]; then
chkconfig $SOLR_SERVICE on
else
update-rc.d $SOLR_SERVICE defaults
fi

service $SOLR_SERVICE start
sleep 5
service $SOLR_SERVICE status

echo "Service $SOLR_SERVICE installed."