WIP: Filter Backends

CHANGELOG.md

@@ -1,8 +1,12 @@
 [unreleased]
 
 * Add testing configuration to `REST_FRAMEWORK` configuration as described in [DRF](https://www.django-rest-framework.org/api-guide/testing/#configuration)
-* Add sorting configuration to `REST_FRAMEWORK` as defined in [json api spec](http://jsonapi.org/format/#fetching-sorting)
 * Add `HyperlinkedRelatedField` and `SerializerMethodHyperlinkedRelatedField`. See [usage docs](docs/usage.md#related-fields)
+* Add optional [jsonapi-style](http://jsonapi.org/format/) filter backends. See [usage docs](docs/usage.md#filter-backends)
+  * query parameter validation -- raises 400 errors rather than silently ignoring "bad" parameters
+  * sort - based on `rest_framework.filters.OrderingFilter`
+  * keyword filter across multiple fields based on `rest_framework.filters.SearchFilter`
+  * field-level filter based on `django_filters.rest_framework.DjangoFilterBackend`
 
 
 v2.5.0 - Released July 11, 2018

README.rst

@@ -173,9 +173,11 @@ override ``settings.REST_FRAMEWORK``
         ),
         'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',
         'DEFAULT_FILTER_BACKENDS': (
-            'rest_framework.filters.OrderingFilter',
+            'rest_framework_json_api.backends.JSONAPIQueryValidationFilter',
+            'rest_framework_json_api.backends.JSONAPIOrderingFilter',
+            'rest_framework_json_api.backends.JSONAPIFilterFilter',
+            'rest_framework_json_api.backends.JSONAPISearchFilter',
         ),
-        'ORDERING_PARAM': 'sort',
         'TEST_REQUEST_RENDERER_CLASSES': (
             'rest_framework_json_api.renderers.JSONRenderer',
         ),

docs/usage.md

@@ -1,10 +1,11 @@
 
 # Usage
 
-The DJA package implements a custom renderer, parser, exception handler, and
+The DJA package implements a custom renderer, parser, exception handler, query filter backends, and
 pagination. To get started enable the pieces in `settings.py` that you want to use.
 
-Many features of the JSON:API format standard have been implemented using Mixin classes in `serializers.py`.
+Many features of the [JSON:API](http://jsonapi.org/format) format standard have been implemented using 
+Mixin classes in `serializers.py`.
 The easiest way to make use of those features is to import ModelSerializer variants
 from `rest_framework_json_api` instead of the usual `rest_framework`
 
@@ -32,9 +33,11 @@ REST_FRAMEWORK = {
     ),
     'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',
     'DEFAULT_FILTER_BACKENDS': (
-        'rest_framework.filters.OrderingFilter',
+        'rest_framework_json_api.backends.JSONAPIQueryValidationFilter',
+        'rest_framework_json_api.backends.JSONAPIOrderingFilter',
+        'rest_framework_json_api.backends.JSONAPIFilterFilter',
+        'rest_framework_json_api.backends.JSONAPISearchFilter',
     ),
-    'ORDERING_PARAM': 'sort',
     'TEST_REQUEST_RENDERER_CLASSES': (
         'rest_framework_json_api.renderers.JSONRenderer',
     ),
@@ -90,6 +93,150 @@ class MyLimitPagination(JsonApiLimitOffsetPagination):
     max_limit = None
 ```
 
+### Filter Backends
+
+[JSON:API](http://jsonapi.org) specifies certain query parameter names but not always the meaning of a particular name.
+The following four filter backends extend existing common filter backends found in DRF and the `django-filter` package
+to comply with the required parts of the [spec](http://jsonapi.org/format) and offer up a Django-flavored
+implementation of parts that are left undefined (like what a `filter` looks like.) The four backends may be
+[configured](http://django-rest-framework.readthedocs.io/en/latest/api-guide/filtering/#setting-filter-backends)
+either as `REST_FRAMWORK['DEFAULT_FILTER_BACKENDS']` or as a list of `filter_backends`. Following is an example
+with explanations of each filter backend after it:
+
+**TODO: change to match example model**
+
+```python
+from rest_framework_json_api.backends import *
+
+class LineItemViewSet(viewsets.ModelViewSet):
+    queryset = LineItem.objects
+    serializer_class = LineItemSerializer
+    filter_backends = (JSONAPIQueryValidationFilter, JSONAPIOrderingFilter, JSONAPIFilterFilter, JSONAPISearchFilter,)
+    filterset_fields = {
+        'subject_area_code': ('exact', 'gt', 'lt',),
+        'course_name': ('exact', 'icontains',),
+        'course_description': ('exact', 'icontains'),
+        'course_identifier': ('exact'),
+        'course_number': ('exact', ),
+        'course_terms__term_identifier': ('exact', 'gt', 'gte', 'lt', 'lte',),
+        'school_bulletin_prefix_code': ('exact', 'regex'),
+    }
+    search_fields = ('course_name', 'course_description', 'course_identifier', 'course_number')
+``` 
+
+#### `JSONAPIQueryValidationFilter`
+`JSONAPIQueryValidationFilter` checks the query parameters to make sure they are all valid per JSON:API
+and returns a `400 Bad Request` if they are not. By default it also flags duplicated `filter` parameters (it is
+generally meaningless to have two of the same filter as filters are ANDed together). You can override these
+attributes if you need to step outside the spec:
+
+**TODO: check this**
+```python
+jsonapi_query_keywords = ('sort', 'filter', 'fields', 'page', 'include')
+jsonapi_allow_duplicated_filters = False
+```
+
+If, for example, your client sends in a query with an invalid parameter (`?sort` misspelled as `?snort`),
+this error will be returned:
+```json
+{
+    "errors": [
+        {
+            "detail": "invalid query parameter: snort",
+            "source": {
+                "pointer": "/data"
+            },
+            "status": "400"
+        }
+    ]
+}
+```
+
+And if two conflicting filters are provided (`?filter[foo]=123&filter[foo]=456`), this error:
+```json
+{
+    "errors": [
+        {
+            "detail": "repeated query parameter not allowed: filter[foo]",
+            "source": {
+                "pointer": "/data"
+            },
+            "status": "400"
+        }
+    ]
+}
+```
+
+If you would rather have your API silently ignore incorrect parameters, simply leave this filter backend out
+and set `jsonapi_allow_duplicated_filters = True`.
+
+#### `JSONAPIOrderingFilter`
+`JSONAPIOrderingFilter` implements the [JSON:API `sort`](http://jsonapi.org/format/#fetching-sorting) just uses
+DRF's [ordering filter](http://django-rest-framework.readthedocs.io/en/latest/api-guide/filtering/#orderingfilter).
+You can use a non-standard parameter name isntead of `sort` by setting `ordering_param`:
+```json
+ordering_param = 'sort'
+jsonapi_ignore_bad_sort_fields = False
+```
+Per the JSON:API, "If the server does not support sorting as specified in the query parameter `sort`,
+it **MUST** return `400 Bad Request`." This error looks like ()for `?sort=`abc,foo,def` where `foo` is a valid
+field name and the other two are not):
+```json
+{
+    "errors": [
+        {
+            "detail": "invalid sort parameters: abc,def",
+            "source": {
+                "pointer": "/data"
+            },
+            "status": "400"
+        }
+    ]
+}
+```
+If you want to silently ignore bad sort fields, set `jsonapi_ignore_bad_sort_fields = True`
+
+#### `JSONAPIFilterFilter`
+`JSONAPIFilterFilter` exploits the power of the [django-filter DjangoFilterBackend](https://django-filter.readthedocs.io/en/latest/guide/rest_framework.html).
+The JSON:API spec explicitly does not define the syntax or meaning of a filter beyond requiring use of the `filter`
+query parameter. This filter implementation is "just a suggestion", but hopefully a useful one with these features:
+- A resource field exact match test: `?filter[foo]=123`
+- Apply other [relational operators](https://docs.djangoproject.com/en/2.0/ref/models/querysets/#field-lookups):
+`?filter[foo.icontains]=bar or ?filter[count.gt]=7...`
+- Membership in a list of values (OR): `?filter[foo]=abc,123,zzz` (foo in ['abc','123','zzz'])
+- Filters can be combined for intersection (AND): `?filter[foo]=123&filter[bar]=abc,123,zzz&filter[...]`
+- Chaining related resource fields for above filters: `?filter[foo.rel.baz]=123` (where `rel` is the relationship name)
+
+Both the usual Django ORM double-underscore notation (`foo__bar__eq`) and a more JSON:API-flavored dotted
+notation (`foo.bar.eq`) are supported.
+
+See the [django-filter](https://django-filter.readthedocs.io/en/latest/guide/rest_framework.html) documentation
+for more details. See an example using a dictionary of filter definitions [above](#filter-backends).
+
+A `400 Bad Request` like the following is returned if the requested filter is not defined:
+```json
+{
+    "errors": [
+        {
+            "detail": "invalid filter[foo]",
+            "source": {
+                "pointer": "/data"
+            },
+            "status": "400"
+        }
+    ]
+}
+```
+
+#### `JSONAPISearchFilter`
+`JSONAPISearchFilter` implements keyword searching across multiple text fields using 
+[`rest_framework.filters.SearchFilter`](http://django-rest-framework.readthedocs.io/en/latest/api-guide/filtering/#searchfilter)
+You configure this filter with `search_fields` and name the filter with `search_param`. For lack of a better name,
+the default is:
+```python
+search_param = 'filter[all]'
+```
+
 ### Performance Testing
 
 If you are trying to see if your viewsets are configured properly to optimize performance,

requirements-development.txt

@@ -1,6 +1,8 @@
 -e .
+django>=2.0
 django-debug-toolbar
 django-polymorphic>=2.0
+django-filter>=2.0
 factory-boy
 Faker
 isort

rest_framework_json_api/backends.py

@@ -0,0 +1,164 @@
+from rest_framework.exceptions import ValidationError
+from rest_framework.filters import BaseFilterBackend, OrderingFilter, SearchFilter
+from rest_framework.settings import api_settings
+
+from django_filters.rest_framework import DjangoFilterBackend
+from rest_framework_json_api.utils import format_value
+
+
+class JSONAPIFilterMixin(object):
+    """
+    class to share data among filtering backends
+    """
+    # search_param is used both in SearchFilter and JSONAPIFilterFilter
+    search_param = api_settings.SEARCH_PARAM
+
+    def __init__(self):
+        self.filter_keys = []
+
+
+class JSONAPIQueryValidationFilter(JSONAPIFilterMixin, BaseFilterBackend):
+    """
+    A backend filter that validates query parameters for jsonapi spec conformance and raises a 400
+    error rather than silently ignoring unknown parameters or incorrect usage.
+
+    set `allow_duplicate_filters = True` if you are OK with the same filter being repeated.
+
+    TODO: For jsonapi error object compliance, must set jsonapi errors "parameter" for the
+          ValidationError. This requires extending DRF/DJA Exceptions.
+    """
+    jsonapi_query_keywords = ('sort', 'filter', 'fields', 'page', 'include')
+    jsonapi_allow_duplicated_filters = False
+
+    def validate_query_params(self, request):
+        """
+        Validate that query params are in the list of valid `jsonapi_query_keywords`
+        Raises ValidationError if not.
+        """
+        for qp in request.query_params.keys():
+            bracket = qp.find('[')
+            if bracket >= 0:
+                if qp[-1] != ']':
+                    raise ValidationError(
+                        'invalid query parameter: {}'.format(qp))
+                keyword = qp[:bracket]
+            else:
+                keyword = qp
+            if keyword not in self.jsonapi_query_keywords:
+                raise ValidationError(
+                    'invalid query parameter: {}'.format(keyword))
+            if not self.jsonapi_allow_duplicated_filters \
+                    and len(request.query_params.getlist(qp)) > 1:
+                raise ValidationError(
+                    'repeated query parameter not allowed: {}'.format(qp))
+
+    def filter_queryset(self, request, queryset, view):
+        self.validate_query_params(request)
+        return queryset
+
+
+class JSONAPIOrderingFilter(JSONAPIFilterMixin, OrderingFilter):
+    """
+    The standard rest_framework.filters.OrderingFilter works mostly fine as is,
+    but with .ordering_param = 'sort'.
+
+    This implements http://jsonapi.org/format/#fetching-sorting and raises 400
+    if any sort field is invalid.
+    """
+    jsonapi_ignore_bad_sort_fields = False
+    ordering_param = 'sort'
+
+    def remove_invalid_fields(self, queryset, fields, view, request):
+        """
+        override remove_invalid_fields to raise a 400 exception instead of silently removing them.
+        """
+        valid_fields = [item[0] for item
+                        in self.get_valid_fields(queryset, view, {'request': request})]
+        bad_terms = [term for term
+                     in fields if format_value(term.lstrip('-'), "underscore") not in valid_fields]
+        if bad_terms and not self.jsonapi_ignore_bad_sort_fields:
+            raise ValidationError(
+                'invalid sort parameter{}: {}'.format(('s' if len(bad_terms) > 1 else ''),
+                                                      ','.join(bad_terms)))
+        return super(JSONAPIOrderingFilter, self).remove_invalid_fields(queryset,
+                                                                        fields, view, request)
+
+
+class JSONAPISearchFilter(JSONAPIFilterMixin, SearchFilter):
+    """
+    The (multi-field) rest_framework.filters.SearchFilter works just fine as is, but with a
+    defined `filter[NAME]` such as `filter[all]` or `filter[_all_]` or something like that.
+
+    This is not part of the jsonapi standard per-se, other than the requirement to use the `filter`
+    keyword: This is an optional implementation of a style of filtering in which a single filter
+    can implement a keyword search across multiple fields of a model as implemented by SearchFilter.
+    """
+    pass
+
+
+class JSONAPIFilterFilter(JSONAPIFilterMixin, DjangoFilterBackend):
+    """
+    Overrides django_filters.rest_framework.DjangoFilterBackend to use `filter[field]` query
+    parameter.
+
+    This is not part of the jsonapi standard per-se, other than the requirement to use the `filter`
+    keyword: This is an optional implementation of style of filtering in which each filter is an ORM
+    expression as implemented by DjangoFilterBackend and seems to be in alignment with an
+    interpretation of http://jsonapi.org/recommendations/#filtering, including relationship
+    chaining.
+
+    Filters can be:
+    - A resource field equality test:
+        `?filter[foo]=123`
+    - Apply other relational operators:
+        `?filter[foo.in]=bar,baz or ?filter[count.ge]=7...`
+    - Membership in a list of values (OR):
+        `?filter[foo]=abc,123,zzz (foo in ['abc','123','zzz'])`
+    - Filters can be combined for intersection (AND):
+        `?filter[foo]=123&filter[bar]=abc,123,zzz&filter[...]`
+    - A related resource field for above tests:
+        `?filter[foo.rel.baz]=123 (where `rel` is the relationship name)`
+
+    It is meaningless to intersect the same filter: ?filter[foo]=123&filter[foo]=abc will
+    always yield nothing so detect this repeated appearance of the same filter in
+    JSONAPIQueryValidationFilter and complain there.
+    """
+
+    def get_filterset(self, request, queryset, view):
+        """
+        Validate that the `filter[field]` is defined in the filters and raise ValidationError if
+        it's missing.
+
+        While `filter` syntax and semantics is undefined by the jsonapi 1.0 spec, this behavior is
+        consistent with the style used for missing query parameters:
+        http://jsonapi.org/format/#query-parameters. In general, unlike django/DRF, jsonapi
+        raises 400 rather than ignoring "bad" query parameters.
+        """
+        fs = super(JSONAPIFilterFilter, self).get_filterset(request, queryset, view)
+        # TODO: change to have option to silently ignore bad filters
+        for k in self.filter_keys:
+            if k not in fs.filters:
+                raise ValidationError("invalid filter[{}]".format(k))
+        return fs
+
+    def get_filterset_kwargs(self, request, queryset, view):
+        """
+        Turns filter[<field>]=<value> into <field>=<value> which is what DjangoFilterBackend expects
+        """
+        self.filter_keys = []
+        # rewrite `filter[field]` query parameters to make DjangoFilterBackend work.
+        data = request.query_params.copy()
+        for qp, val in data.items():
+            if qp[:7] == 'filter[' and qp[-1] == ']' and qp != self.search_param:
+                # convert jsonapi relationship path to Django ORM's __ notation:
+                key = qp[7:-1].replace('.', '__')
+                # undo JSON_API_FORMAT_FIELD_NAMES conversion:
+                key = format_value(key, 'underscore')
+                data[key] = val
+                self.filter_keys.append(key)
+                del data[qp]
+        return {
+            'data': data,
+            'queryset': queryset,
+            'request': request,
+        }