在移动设备上进行全文搜索？

我们很快就会着手开发新的移动应用程序.此特定应用程序将用于大量搜索基于文本的字段.整个集团对于什么类型的数据库引擎最适合在移动平台上允许这些类型的搜索的任何建议？

具体包括Windows Mobile 6,我们将使用.Net CF. 此外,一些基于文本的字段将在35到500个字符之间.该设备将以两种不同的方式运行,批量和WiFi.当然,对于WiFi,我们只需向完整的数据库引擎提交请求,只需获取结果即可.这个问题围绕着"批处理"版本,该版本将容纳一个装有闪存/可移动存储卡设备信息的数据库.

无论如何,我知道SQLCE有一些基本的索引,但你没有进入真正花哨的"全文"样式索引,直到你有一个完整的版本,当然在移动平台上不可用.

数据外观的一个示例:

"围裙木匠可调节皮革腰包腰部五金腰带"等

我还没有进入任何其他特定选项的评估,因为我认为我会利用这个小组的经验来首先指出一些特定的途径.

有什么建议/提示吗？

就在最近,我遇到了同样的问题.这是我做的:

我创建了一个类来保存每个对象的id和文本(在我的例子中,我称之为sku(项目编号)和描述).这会创建一个较小的对象,使用较少的内存,因为它仅用于搜索.找到匹配后,我仍然会从数据库中获取完整的对象.

public class SmallItem
{
    private int _sku;
    public int Sku
    {
        get { return _sku; }
        set { _sku = value; }
    }

    // Size of max description size + 1 for null terminator.
    private char[] _description = new char[36];
    public char[] Description
    {
        get { return _description; }
        set { _description = value; }
    }

    public SmallItem()
    {
    }
}

创建此类之后,您可以创建这些对象的数组(我实际上在我的情况下使用了List)并使用它在整个应用程序中进行搜索.此列表的初始化需要一些时间,但您只需要在启动时担心这一点.基本上只需对您的数据库运行查询并获取创建此列表所需的数据.

获得列表后,您可以快速浏览它,搜索您想要的任何单词.由于它是一个包含,它还必须在单词中找到单词(例如,钻头将返回钻孔,钻头,钻头等).为此,我们编写了一个自行开发的非托管c#包含函数.它接受一个字符串数组(因此您可以搜索多个单词...我们将其用于"AND"搜索...描述必须包含传入的所有单词..."OR"目前不受支持在这个例子中).当它搜索单词列表时,它会构建一个ID列表,然后将其传递回调用函数.获得ID列表后,您可以在数据库中轻松运行快速查询,以根据快速索引的ID号返回完整的对象.我应该提一下,我们还限制了返回的最大结果数.这可以拿出来.如果有人输入"e"这样的搜索词,那就太方便了.这将带来很多结果.

以下是自定义包含函数的示例:

public static int[] Contains(string[] descriptionTerms, int maxResults, List itemList)
{
    // Don't allow more than the maximum allowable results constant.            
    int[] matchingSkus = new int[maxResults];

    // Indexes and counters.
    int matchNumber = 0;
    int currentWord = 0;
    int totalWords = descriptionTerms.Count() - 1;  // - 1 because it will be used with 0 based array indexes

    bool matchedWord;

    try
    {   
        /* Character array of character arrays. Each array is a word we want to match.
         * We need the + 1 because totalWords had - 1 (We are setting a size/length here,
         * so it is not 0 based... we used - 1 on totalWords because it is used for 0
         * based index referencing.)
         * */
        char[][] allWordsToMatch = new char[totalWords + 1][];

        // Character array to hold the current word to match. 
        char[] wordToMatch = new char[36]; // Max allowable word size + null terminator... I just picked 36 to be consistent with max description size.

        // Loop through the original string array or words to match and create the character arrays. 
        for (currentWord = 0; currentWord <= totalWords; currentWord++)
        {
            char[] desc = new char[descriptionTerms[currentWord].Length + 1];
            Array.Copy(descriptionTerms[currentWord].ToUpper().ToCharArray(), desc, descriptionTerms[currentWord].Length);
            allWordsToMatch[currentWord] = desc;
        }

        // Offsets for description and filter(word to match) pointers.
        int descriptionOffset = 0, filterOffset = 0;

        // Loop through the list of items trying to find matching words.
        foreach (SmallItem i in itemList)
        {
            // If we have reached our maximum allowable matches, we should stop searching and just return the results.
            if (matchNumber == maxResults)
                break;

            // Loop through the "words to match" filter list.
            for (currentWord = 0; currentWord <= totalWords; currentWord++)
            {
                // Reset our match flag and current word to match.
                matchedWord = false;
                wordToMatch = allWordsToMatch[currentWord];

                // Delving into unmanaged code for SCREAMING performance ;)
                unsafe
                {
                    // Pointer to the description of the current item on the list (starting at first char).
                    fixed (char* pdesc = &i.Description[0])
                    {
                        // Pointer to the current word we are trying to match (starting at first char).
                        fixed (char* pfilter = &wordToMatch[0])
                        {
                            // Reset the description offset.
                            descriptionOffset = 0;

                            // Continue our search on the current word until we hit a null terminator for the char array.
                            while (*(pdesc + descriptionOffset) != '\0')
                            {
                                // We've matched the first character of the word we're trying to match.
                                if (*(pdesc + descriptionOffset) == *pfilter)
                                {
                                    // Reset the filter offset.
                                            filterOffset = 0;

                                    /* Keep moving the offsets together while we have consecutive character matches. Once we hit a non-match
                                     * or a null terminator, we need to jump out of this loop.
                                     * */
                                    while (*(pfilter + filterOffset) != '\0' && *(pfilter + filterOffset) == *(pdesc + descriptionOffset))
                                    {
                                        // Increase the offsets together to the next character.
                                        ++filterOffset;
                                        ++descriptionOffset;
                                    }

                                    // We hit matches all the way to the null terminator. The entire word was a match.
                                    if (*(pfilter + filterOffset) == '\0')
                                    {
                                        // If our current word matched is the last word on the match list, we have matched all words.
                                        if (currentWord == totalWords)
                                        {
                                            // Add the sku as a match.
                                            matchingSkus[matchNumber] = i.Sku.ToString();
                                            matchNumber++;

                                            /* Break out of this item description. We have matched all needed words and can move to
                                             * the next item.
                                             * */
                                            break;
                                        }

                                        /* We've matched a word, but still have more words left in our list of words to match.
                                         * Set our match flag to true, which will mean we continue continue to search for the
                                         * next word on the list.
                                         * */
                                         matchedWord = true;
                                    }
                                }

                                // No match on the current character. Move to next one.
                                descriptionOffset++;
                            }

                            /* The current word had no match, so no sense in looking for the rest of the words. Break to the
                             * next item description.
                             * */
                             if (!matchedWord)
                                break;
                        }
                    }
                }
            }
        };

        // We have our list of matching skus. We'll resize the array and pass it back.
        Array.Resize(ref matchingSkus, matchNumber);
        return matchingSkus;
    }
    catch (Exception ex)
    {
        // Handle the exception
    }
}

获得匹配的skus列表后,可以遍历数组并构建一个仅返回匹配skus的查询命令.

对于性能的概念,这是我们发现的(执行以下步骤):

搜索~171,000项

创建所有匹配项的列表

查询数据库,仅返回匹配的项目

构建完整的项目(类似于SmallItem类,但是更多的字段)

使用完整项目对象填充数据网格.

在我们的移动设备上,整个过程需要2-4秒(如果我们在搜索所有项目之前达到匹配限制则需要2秒...如果我们必须扫描每个项目需要4秒钟).

我也尝试过这样做而没有非托管代码并使用String.IndexOf(并尝试过String.Contains ......具有与IndexOf相同的性能).那样慢得多......大约25秒.

我也尝试过使用StreamReader和包含[Sku Number] | [Description]行的文件.代码类似于非托管代码示例.整个扫描过程大约需要15秒.速度不是太差,但不是很好.文件和StreamReader方法比我向您展示的方式有一个优势.该文件可以提前创建.我向您展示的方式需要内存和应用程序启动时加载List的初始时间.对于我们的171,000件商品,这大约需要2分钟.如果你能够在每次应用程序启动时等待初始加载(当然可以在单独的线程上完成),那么以这种方式搜索是最快的方式(我至少找到了).

希望有所帮助.

PS - 感谢Dolch帮助处理一些非托管代码.