| 1 |
====== |
| 2 |
Search |
| 3 |
====== |
| 4 |
|
| 5 |
A common task for web applications is to search some data in the database with |
| 6 |
user input. In a simple case, this could be filtering a list of objects by a |
| 7 |
category. A more complex use case might require searching with weighting, |
| 8 |
categorization, highlighting, multiple languages, and so on. This document |
| 9 |
explains some of the possible use cases and the tools you can use. |
| 10 |
|
| 11 |
We'll refer to the same models used in :doc:`/topics/db/queries`. |
| 12 |
|
| 13 |
Use Cases |
| 14 |
========= |
| 15 |
|
| 16 |
Standard textual queries |
| 17 |
------------------------ |
| 18 |
|
| 19 |
Text-based fields have a selection of matching operations. For example, you may |
| 20 |
wish to allow lookup up an author like so: |
| 21 |
|
| 22 |
.. code-block:: pycon |
| 23 |
|
| 24 |
>>> Author.objects.filter(name__contains="Terry") |
| 25 |
[<Author: Terry Gilliam>, <Author: Terry Jones>] |
| 26 |
|
| 27 |
This is a very fragile solution as it requires the user to know an exact |
| 28 |
substring of the author's name. A better approach could be a case-insensitive |
| 29 |
match (:lookup:`icontains`), but this is only marginally better. |
| 30 |
|
| 31 |
A database's more advanced comparison functions |
| 32 |
----------------------------------------------- |
| 33 |
|
| 34 |
If you're using PostgreSQL, Django provides :doc:`a selection of database |
| 35 |
specific tools </ref/contrib/postgres/search>` to allow you to leverage more |
| 36 |
complex querying options. Other databases have different selections of tools, |
| 37 |
possibly via plugins or user-defined functions. Django doesn't include any |
| 38 |
support for them at this time. We'll use some examples from PostgreSQL to |
| 39 |
demonstrate the kind of functionality databases may have. |
| 40 |
|
| 41 |
.. admonition:: Searching in other databases |
| 42 |
|
| 43 |
All of the searching tools provided by :mod:`django.contrib.postgres` are |
| 44 |
constructed entirely on public APIs such as :doc:`custom lookups |
| 45 |
</ref/models/lookups>` and :doc:`database functions |
| 46 |
</ref/models/database-functions>`. Depending on your database, you should |
| 47 |
be able to construct queries to allow similar APIs. If there are specific |
| 48 |
things which cannot be achieved this way, please open a ticket. |
| 49 |
|
| 50 |
In the above example, we determined that a case insensitive lookup would be |
| 51 |
more useful. When dealing with non-English names, a further improvement is to |
| 52 |
use :lookup:`unaccented comparison <unaccent>`: |
| 53 |
|
| 54 |
.. code-block:: pycon |
| 55 |
|
| 56 |
>>> Author.objects.filter(name__unaccent__icontains="Helen") |
| 57 |
[<Author: Helen Mirren>, <Author: Helena Bonham Carter>, <Author: Hélène Joy>] |
| 58 |
|
| 59 |
This shows another issue, where we are matching against a different spelling of |
| 60 |
the name. In this case we have an asymmetry though - a search for ``Helen`` |
| 61 |
will pick up ``Helena`` or ``Hélène``, but not the reverse. Another option |
| 62 |
would be to use a :lookup:`trigram_similar` comparison, which compares |
| 63 |
sequences of letters. |
| 64 |
|
| 65 |
For example: |
| 66 |
|
| 67 |
.. code-block:: pycon |
| 68 |
|
| 69 |
>>> Author.objects.filter(name__unaccent__lower__trigram_similar="Hélène") |
| 70 |
[<Author: Helen Mirren>, <Author: Hélène Joy>] |
| 71 |
|
| 72 |
Now we have a different problem - the longer name of "Helena Bonham Carter" |
| 73 |
doesn't show up as it is much longer. Trigram searches consider all |
| 74 |
combinations of three letters, and compares how many appear in both search and |
| 75 |
source strings. For the longer name, there are more combinations that don't |
| 76 |
appear in the source string, so it is no longer considered a close match. |
| 77 |
|
| 78 |
The correct choice of comparison functions here depends on your particular data |
| 79 |
set, for example the language(s) used and the type of text being searched. All |
| 80 |
of the examples we've seen are on short strings where the user is likely to |
| 81 |
enter something close (by varying definitions) to the source data. |
| 82 |
|
| 83 |
Document-based search |
| 84 |
--------------------- |
| 85 |
|
| 86 |
Standard database operations stop being a useful approach when you start |
| 87 |
considering large blocks of text. Whereas the examples above can be thought of |
| 88 |
as operations on a string of characters, full text search looks at the actual |
| 89 |
words. Depending on the system used, it's likely to use some of the following |
| 90 |
ideas: |
| 91 |
|
| 92 |
- Ignoring "stop words" such as "a", "the", "and". |
| 93 |
- Stemming words, so that "pony" and "ponies" are considered similar. |
| 94 |
- Weighting words based on different criteria such as how frequently they |
| 95 |
appear in the text, or the importance of the fields, such as the title or |
| 96 |
keywords, that they appear in. |
| 97 |
|
| 98 |
There are many alternatives for using searching software, some of the most |
| 99 |
prominent are Elastic_ and Solr_. These are full document-based search |
| 100 |
solutions. To use them with data from Django models, you'll need a layer which |
| 101 |
translates your data into a textual document, including back-references to the |
| 102 |
database ids. When a search using the engine returns a certain document, you |
| 103 |
can then look it up in the database. There are a variety of third-party |
| 104 |
libraries which are designed to help with this process. |
| 105 |
|
| 106 |
.. _Elastic: https://www.elastic.co/ |
| 107 |
.. _Solr: https://solr.apache.org/ |
| 108 |
|
| 109 |
PostgreSQL support |
| 110 |
~~~~~~~~~~~~~~~~~~ |
| 111 |
|
| 112 |
PostgreSQL has its own full text search implementation built-in. While not as |
| 113 |
powerful as some other search engines, it has the advantage of being inside |
| 114 |
your database and so can easily be combined with other relational queries such |
| 115 |
as categorization. |
| 116 |
|
| 117 |
The :mod:`django.contrib.postgres` module provides some helpers to make these |
| 118 |
queries. For example, a query might select all the blog entries which mention |
| 119 |
"cheese": |
| 120 |
|
| 121 |
.. code-block:: pycon |
| 122 |
|
| 123 |
>>> Entry.objects.filter(body_text__search="cheese") |
| 124 |
[<Entry: Cheese on Toast recipes>, <Entry: Pizza recipes>] |
| 125 |
|
| 126 |
You can also filter on a combination of fields and on related models: |
| 127 |
|
| 128 |
.. code-block:: pycon |
| 129 |
|
| 130 |
>>> Entry.objects.annotate( |
| 131 |
... search=SearchVector("blog__tagline", "body_text"), |
| 132 |
... ).filter(search="cheese") |
| 133 |
[ |
| 134 |
<Entry: Cheese on Toast recipes>, |
| 135 |
<Entry: Pizza Recipes>, |
| 136 |
<Entry: Dairy farming in Argentina>, |
| 137 |
] |
| 138 |
|
| 139 |
See the ``contrib.postgres`` :doc:`/ref/contrib/postgres/search` document for |
| 140 |
complete details. |