phpで文章の重複率を取得する その1
phpで文章の重複率を取得する方法
1.AとBの文章があった場合にどれだけ文章が重複しているかを%で出す
2.コンテンツ的な重複なので、完全一致でなく、内容も鑑みる。
3.文章の前後が違うだけで、文節が一緒の場合も重複とみなす。
考えられる対応策
1.日本語文章を文節で分ける
2.A/Bの文章の要素を分けた文節を、array_diff で差分をとる。
3.A/Bの文章の要素が少ないほうを本体とする(とりあえず)
4.count(差分)/ count(本体) * 100 = 重複率の%
と考えてみた。
http://mecab.sourceforge.net/#download
mecab-0.96.tar.gzをいんすこ
% tar zxfv mecab-X.X.tar.gz
% cd mecab-X.X
% ./configure
% make
% make check
% su
# make install
辞書もインスコ
% tar zxfv mecab-ipadic-2.7.0-XXXX.tar.gz
% mecab-ipadic-2.7.0-XXXX
% ./configure
% make
% su
# make install
php_mecab-0.2.0.tgz (alpha)
http://page2.xrea.jp/pub/php_mecab-0.2.0.tgz
$ ./configure --with-php-config=/usr/bin/php-config --with-mecab=/usr/local/bin/mecab-config
configure: WARNING: You will need re2c 0.9.11 or later if you want to regenerate PHP parsers.
とでるが問題ないみたいなので無視
$ make
$ su
# make install
ここでLD_PATH?何ちゃらの警告がでる。
そこで、/etc/lo.so.confに追加
/usr/lib64/php/modules/
そのあとに
/sbin/ldconfig
最後に/etc/php.iniの最末尾に
このあと、
/etc/rc.d/init.d/httpd reload
mecab
MeCab Support => enabled
MeCab Library Version => 0.96
Module Version => 0.2.0-alpha
があるのを確認
そして、本題に入ります。
本題は次回で。。。
php-eaccelerator.x86_64 をインスコする
Turck MMCache時代からずっと愛用しているので、早速インストール。
PHP動かすなら、絶対に外せない1本!
PVあるサイトならディスクI/Oが大幅に減るので、HDDにもやさしい。
php-eaccelerator.x86_64をCentos4.4 にインストールする
# yum search eaccelerator
Searching Packages:
Setting up repositories
update 100% |=========================| 951 B 00:00
base 100% |=========================| 1.1 kB 00:00
addons 100% |=========================| 951 B 00:00
extras 100% |=========================| 1.1 kB 00:00
Reading repository metadata in from local files
php-eaccelerator.x86_64 5.1.6_0.9.5-1.el4.cent
Matched from:
php-eaccelerator
eAccelerator is a further development of the MMCache PHP Accelerator & Encoder.
It increases performance of PHP scripts by caching them in compiled state, so
that the overhead of compiling is almost completely eliminated.
http://eaccelerator.sourceforge.net/
# yum -y install php-eaccelerator.x86_64
そのあと、
vi /etc/php.d/eaccelerator.ini
[eaccelerator]
;;;;;;;;;;;;;;;;;;;
; About this file ;
;;;;;;;;;;;;;;;;;;;
;
; eAccelerator is compatible with Zend Optimizer's loader. Zend Optimizer
; must be installed after eAccelerator in php.ini. If you don't use scripts
; encoded with Zend Encoder then we do not recommend you install Zend Optimizer
; with eAccelerator.
; You must uncomment one (and only one) line from the following to load
; eAccelerator extension.
;extension="eaccelerator.so"
zend_extension="/usr/lib64/php/modules/eaccelerator.so"
;zend_extension_ts="/usr/lib64/php/modules/eaccelerator.so"
; The amount of shared memory (in megabytes) that eAccelerator will use.
; "0" means OS default. Default value is "0".
eaccelerator.shm_size = "48" ;※キャッシュしたいphpソースの全体の大きさに合わせる
; The directory that is used for disk cache. eAccelerator stores precompiled
; code, session data, content and user entries here. The same data can be
; stored in shared memory also (for more quick access). Default value is
; "/var/cache/php-eaccelerator".
eaccelerator.cache_dir = "/var/cache/php-eaccelerator"
; Enables or disables eAccelerator. Should be "1" for enabling or
; "0" for disabling. Default value is "1".
eaccelerator.enable = "1"
; Enables or disables internal peephole optimizer which may speed up code
; execution. Should be "1" for enabling or "0" for disabling.
; Default value is "1".
eaccelerator.optimizer = "1"
; Enables or disables debug logging. Setting this to 1 will print information
; to the log file about the cach hits of a file.
eaccelerator.debug = 0
; Set the log file for eaccelerator. When this option isn't set then the data
; will be logged to stderr
eaccelerator.log_file = "/var/log/httpd/eaccelerator_log"
; A string that's prepended to all keys. This allows two applications that use the
; same key names to run on the same host by setting this in .htaccess or in the main
; configuration file for the whole webserver.
eaccelerator.name_space = ""
; Enables or disables PHP file modification checking. Should be "1"
; for enabling or "0" for disabling. You should set it to "1" if you want
; to recompile PHP files after modification. Default value is "1".
eaccelerator.check_mtime = "1"
; Determine which PHP files must be cached. You may specify the number of
; patterns (for example "*.php *.phtml") which specifies to cache or
; not to cache. If pattern starts with the character "!", it means to ignore
; files which are matched by the following pattern. Default value is "" that
; means - all PHP scripts will be cached.
eaccelerator.filter = ""
; Disables putting large values into shared memory by "eaccelerator_put()"
; function.
; It indicates the largest allowed size in bytes (10240, 10K, 1M). The "0"
; disables the limit. Default value is "0".
eaccelerator.shm_max = "0"
; When eAccelerator fails to get shared memory for new script it removes
; all scripts which were not accessed at last "shm_ttl" seconds from shared
; memory. Default value is "0" that means - don't remove any files from
; shared memory.
eaccelerator.shm_ttl = "3600"
; When eAccelerator fails to get shared memory for new script it tryes to
; remove old script if the previous try was made more then "shm_prune_period"
; seconds ago. Default value is "0" that means - don't try to remove any
; files from shared memory.
eaccelerator.shm_prune_period = "0"
; Enables or disables caching of compiled scripts on disk. It has no effect
; on session data and content caching.
; Default value is "0" that means - use disk and shared memory for caching.
eaccelerator.shm_only = "0"
; Enables or disables cached content compression. Default value is "1" that
; means enable compression.
eaccelerator.compress = "1"
; Compression level used for content caching. Default value is "9" which i
; is the maximum compression level
eaccelerator.compress_level = "9"
; Determine where keys, session data and content will be cached. The possible
; values are:
; "shm_and_disk" - cache data in shared memory and on disk (default value)
; "shm" - cache data in shared memory or on disk if shared memory
; is full or data size greater then "eaccelerator.shm_max"
; "shm_only" - cache data in shared memory
; "disk_only" - cache data on disk
; "none" - don't cache data
eaccelerator.keys = "shm_and_disk"
eaccelerator.sessions = "shm_and_disk"
eaccelerator.content = "shm_and_disk"
; The script paths that are allowed to get admin information and do admin
; controls
eaccelerator.allowed_admin_path = "/home/hoge/hoge/" ;※admin画面を表示させる場所を決める
最後に
/etc/rc.d/init.d/httpd restart
php -v で
PHP 5.1.6 (cli)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
with eAccelerator v0.9.5, Copyright (c) 2004-2006 eAccelerator, by eAccelerator
がでればOK