构建的grep比Linux附带的grep慢

我试图理解为什么我构建的grep比系统附带的grep慢得多,并试图找到系统附带的grep使用的编译器选项.

操作系统版本:CentOS版本5.3(最终)grep系统:

  Version: grep (GNU grep) 2.5.1
  Size: 88896 bytes
  ldd output: 
 libpcre.so.0 => /lib64/libpcre.so.0 (0x0000003991800000)
 libc.so.6 => /lib64/libc.so.6 (0x0000003985a00000)
 /lib64/ld-linux-x86-64.so.2 (0x0000003984a00000)

grep由我建造:

  Version: 2.5.1
  Size: 256437 bytes
  ldd output:
 libpcre.so.0 => /lib64/libpcre.so.0 (0x0000003991800000)
 libc.so.6 => /lib64/libc.so.6 (0x0000003985a00000)
 /lib64/ld-linux-x86-64.so.2 (0x0000003984a00000)

在大型列表文本文件上运行正则表达式搜索时,系统grep(330 msecs)的性能比我构建的grep(22430 msecs)快得多.

以下是我以前的时间命令..

% time src/grep ".*asa.*" large_list.txt > /dev/null
real 0m22.430s
user 0m22.291s
sys 0m0.080s

要么

% time bin/grep ".*asa.*" large_list.txt > /dev/null
real 0m0.331s
user 0m0.236s
sys 0m0.081s

系统grep显然正在使用一些可提供巨大性能差异的优化选项.

有些人可以帮我解决系统grep可能构建的选项吗？

这是我构建时其中一个源文件的编译选项..
gcc -DLIBDIR=\"/usr/local/lib\" -DHAVE_CONFIG_H -I. -I.. -I.. -I. -I../intl -g -O2 -MT xstrtol.o -MD -MP -MF .deps/xstrtol.Tpo -c -o xstrtol.o xstrtol.c

./configure的输出:

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gawk... (cached) gawk
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for a BSD-compatible install... /usr/bin/install -c
checking for ranlib... ranlib
checking for getconf... getconf
checking for CFLAGS value to request large file support... 
checking for LDFLAGS value to request large file support... 
checking for LIBS value to request large file support... 
checking for _FILE_OFFSET_BITS... no
checking for _LARGEFILE_SOURCE... no
checking for _LARGE_FILES... no
checking for function prototypes... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for string.h... (cached) yes
checking for size_t... yes
checking for ssize_t... yes
checking for an ANSI C-conforming const... yes
checking for inttypes.h... yes
checking for unsigned long long... yes
checking for ANSI C header files... (cached) yes
checking for string.h... (cached) yes
checking for stdlib.h... (cached) yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking for memory.h... (cached) yes
checking for unistd.h... (cached) yes
checking libintl.h usability... yes
checking libintl.h presence... yes
checking for libintl.h... yes
checking wctype.h usability... yes
checking wctype.h presence... yes
checking for wctype.h... yes
checking wchar.h usability... yes
checking wchar.h presence... yes
checking for wchar.h... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking whether stat file-mode macros are broken... no
checking for working alloca.h... yes
checking for alloca... yes
checking whether closedir returns void... no
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for getpagesize... yes
checking for working mmap... yes
checking for btowc... yes
checking for isascii... yes
checking for iswctype... yes
checking for mbrlen... yes
checking for memmove... yes
checking for setmode... no
checking for strerror... yes
checking for wcrtomb... yes
checking for wcscoll... yes
checking for wctype... yes
checking whether mbrtowc and mbstate_t are properly declared... yes
checking for stdlib.h... (cached) yes
checking for mbstate_t... yes
checking for memchr... yes
checking for stpcpy... yes
checking for strtoul... yes
checking for atexit... yes
checking for fnmatch... yes
checking for stdlib.h... (cached) yes
checking whether  defines strtoumax as a macro... no
checking for strtoumax... yes
checking whether strtoul is declared... yes
checking whether strtoull is declared... yes
checking for strerror in -lcposix... no
checking for inline... inline
checking for off_t... yes
checking whether we are using the GNU C Library 2.1 or newer... yes
checking argz.h usability... yes
checking argz.h presence... yes
checking for argz.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking nl_types.h usability... yes
checking nl_types.h presence... yes
checking for nl_types.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... (cached) yes
checking for feof_unlocked... yes
checking for fgets_unlocked... yes
checking for getcwd... yes
checking for getegid... yes
checking for geteuid... yes
checking for getgid... yes
checking for getuid... yes
checking for mempcpy... yes
checking for munmap... yes
checking for putenv... yes
checking for setenv... yes
checking for setlocale... yes
checking for stpcpy... (cached) yes
checking for strchr... yes
checking for strcasecmp... yes
checking for strdup... yes
checking for strtoul... (cached) yes
checking for tsearch... yes
checking for __argz_count... yes
checking for __argz_stringify... yes
checking for __argz_next... yes
checking for iconv... yes
checking for iconv declaration... 
         extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
checking for nl_langinfo and CODESET... yes
checking for LC_MESSAGES... yes
checking whether NLS is requested... yes
checking whether included gettext is requested... no
checking for libintl.h... (cached) yes
checking for GNU gettext in libc... yes
checking for dcgettext... yes
checking for msgfmt... /usr/bin/msgfmt
checking for gmsgfmt... /usr/bin/msgfmt
checking for xgettext... /usr/bin/xgettext
checking for bison... bison
checking version of bison... 2.3, ok
checking for catalogs to be installed...  af be bg ca cs da de el eo es et eu fi fr ga gl he hr hu id it ja ko ky lt nb nl pl pt pt_BR ro ru rw sk sl sr sv tr uk vi zh_TW
checking for dos file convention... no
checking host system type... (cached) x86_64-unknown-linux-gnu
checking host system type... (cached) x86_64-unknown-linux-gnu
checking for DJGPP environment... no
checking for environ variable separator... :
checking for working re_compile_pattern... yes
checking for getopt_long... yes
configure: WARNING: Included lib/regex.c not used
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... no
checking for strerror... (cached) yes
checking for strerror_r... (cached) yes
checking for vprintf... yes
checking for doprnt... no
checking for ANSI C header files... (cached) yes
checking for working malloc... yes
checking for working realloc... yes
checking for pcre_exec in -lpcre... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating lib/Makefile
config.status: creating lib/posix/Makefile
config.status: creating src/Makefile
config.status: creating tests/Makefile
config.status: creating po/Makefile.in
config.status: creating intl/Makefile
config.status: WARNING:  intl/Makefile.in seems to ignore the --datarootdir setting
config.status: creating doc/Makefile
config.status: creating m4/Makefile
config.status: creating vms/Makefile
config.status: creating bootstrap/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing default-1 commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
config.status: executing stamp-h commands

谢谢,库马尔

为什么不为grep二进制文件获取CentOS的SRPM并将它们的编译选项与你的比较？我猜这比让整个StackOverflow社区盲目地在黑暗中探索直到他们遇到某些东西要高效得多.

编辑:您使用多字节编码的语言环境？(注意:如果您不知道这意味着什么,那么答案可能是"是",因为UTF-8已经成为大多数Linux发行版的默认值,而且RedHat(以及CentOS)确实是第一个做开关).

在那种情况下,GNU grep 是狗慢.这不仅适用于GNU grep,而且适用于几乎所有进行某种文本处理的GNU工具.FSF拒绝接受任何补丁来提高多字节性能,除非这些补丁被证明不会减慢固定宽度编码.但是,由于任何用于提高多字节编码性能的补丁必须至少在if某处包含某些语句,因此实际上不可能编写一个补丁,该补丁至少不会使固定宽度编码减慢该if语句的开销.因此,GNU工具的UTF-8性能将持续下降,直到时间结束.

总之,大多数Linux发行商不给老鼠的寻呼自由软件基金会的想法,反正修补GNU的grep.在Fedora的罗海德SRPM包含一个名为补丁grep-2.5.3-egf-speedup.patch,这加速了几个数量级的grep GNU的UTF-8性能.(由于这个补丁已经从2005年开始,我认为它也在CentOS中使用.)这个补丁也用在Mac OSX,Debian,Ubuntu ......中,几乎没有人使用GNU grep作为GNU分发的.多字节编码中的文本处理永远不会像固定宽度编码那样快,但它至少应该是可比较的,而不是50倍(甚至有些人报告的1500倍).

还有另一个叫做的补丁dfa-optional,它使grep只使用GNU libc的正则表达式引擎而不是它自己的,这不仅在处理UTF-8时要快得多,而且bug也少得多.

因此,您可能希望使用export LC_ALL=POSIXset 重新运行基准测试.如果这可以解决您的问题,则需要应用上述两个补丁中的任何一个.

这两个RedHat错误报告中还提供了更多信息:

错误69900 - grep写入输出非常慢

错误121313 - 多字节LC_CTYPE上的grep SLOW

故事的寓意:尽管普遍认为,Linux发行商确实知道他们在做什么,至少有时候.不要猜测它们.