Gitingest

Include files under: 50kb

Summary

Repository: rfyiamcool/elasticsearch_parse
Files analyzed: 14

Estimated tokens: 16.3k

Download

Directory Structure

Directory structure:

└── rfyiamcool-elasticsearch_parse/

    ├── README.md

    ├── LICENSE.txt

    ├── setup.py

    ├── test.py

    └── elasticsearch_parse/

        ├── __init__.py

        ├── aggs.py

        ├── exceptions.py

        ├── faceted_search.py

        ├── filter.py

        ├── index.py

        ├── query.py

        ├── search.py

        ├── serializer.py

        └── utils.py

Files Content

================================================
FILE: README.md
================================================
# elasticsearch_parse
=================

首先注明下本项目是由来,学习Elasticsearch本身是有些痛苦的, 我们可以熟练的编写SQL 、 Mongodb语句，但对于Elasticsearch Dsl语法总是蒙头，一个劲的翻找笔记...  
简单说 Elasticsearch Parse可以让你更容易的上手ES, 他的功能主要是语句映射, 有点ORM的意思... 

此项目继承于`elasticsearch-dsl-py`,本来是想fork子项目,开发好后提交pull request, 但是想到我这语法映射的功能, 对于官方来说不是很稀罕,索性直接砍掉了 70% 代码,然后中间又加了一些佐料, 最终只是为了DSL语句映射功能 ! 

模块安装方法:
```
pip install elasticsearch_parse
```

下面我们来体验下封装后的es语法解释器.

首先用原始的DSL语法操作,一眼望去会有些麻烦, 手写起来会更麻烦.  
```
from elasticsearch import Elasticsearch
client = Elasticsearch()

response = client.search(
    index="my-index",
    body={
      "query": {
        "filtered": {
          "query": {
            "bool": {
              "must": [{"match": {"title": "python"}}],
              "must_not": [{"match": {"description": "beta"}}]
            }
          },
          "filter": {"term": {"category": "search"}}
        }
      },
      "aggs" : {
        "per_tag": {
          "terms": {"field": "tags"},
          "aggs": {
            "max_lines": {"max": {"field": "lines"}}
          }
        }
      }
    }
)

```

下面是使用Elasticsearch_parse的用法,要多简单就多简单

```
from elasticsearch_parse import Search, Q

s = Search(index="my-index") \
    .filter("term", blog="xiaorui.cc") \
    .query("match", author="ruifengyun")   \
    .query(~Q("match", face="good"))

s.aggs.bucket('per_tag', 'terms', field='tags')

response = s.execute()
```
我们得到的结果是:
```
{
    "query": {
        "filtered": {
            "filter": {
                "term": {
                    "blog": "xiaorui.cc"
                }
            },
            "query": {
                "bool": {
                    "must_not": [
                        {
                            "match": {
                                "face": "good"
                            }
                        }
                    ],
                    "must": [
                        {
                            "match": {
                                "author": "ruifengyun"
                            }
                        }
                    ]
                }
            }
        }
    },
    "aggs": {
        "per_tag": {
            "terms": {
                "field": "tags"
            }
        }
    }
}
```

我们把语法的用法给过一遍.
```
s = search.Search()
```

通过match查询,f字段值为55的数据
```
s.query('match', f=55)
```

时间范围
```
s.query('range', ** {'@timestamp': {'lt': 'now'}})
```

外围的size的控制  
```
s = s.query('match', f=42)
s[3].to_dict() {'query': {'match_all': {}}, 'from': 3, 'size': 1}
```

```
assert s.to_dict(size=10) == {"query": {"match": {'f': 42}}, "size": 10}
```

嵌入内部size控制
```
s = search.Search.from_dict({"size": 5})
assert {
     "query": {"match_all": {}},
     "size": 5
} == s.to_dict()
```

对于aggs的聚合的使用
```
s = s.query('match', f=42)
assert {"query": {"match": {'f': 42}}} == s.to_dict()
assert {"query": {"match": {'f': 42}}, "size": 10} == s.to_dict(size=10)
s.aggs.bucket('per_tag', 'terms', field='f').metric('max_score', 'max', field='score')
d = {
    'aggs': {
        'per_tag': {
            'terms': {'field': 'f'},
            'aggs': {'max_score': {'max': {'field': 'score'}}}
         }
    }
```
下面在介绍下aggs的用法:

取出某个字段最大的数据
```
s.aggs.Max(field='score')
```

按照条件分组聚合
```
s.aggs.A('terms', field='tags', aggs={'max_score': max_score})
```

分组聚合,Elasticsearch会帮你针对count排序
```
s.aggs.bucket('per_tag', 'terms', field='tags')
```


...





================================================
FILE: LICENSE.txt
================================================
The MIT License (MIT)

Copyright (c) 2015 rui fengyun

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



================================================
FILE: setup.py
================================================
#!/usr/bin/env python

#from distutils.core import setup, Command
from setuptools import setup,find_packages
import os
import os.path

def read(fname):
    return open(os.path.join(os.path.dirname(__file__), fname)).read()

setup(
    name='elasticsearch_parse',
    version='2.0',
    description='elasticsearch parse',
    long_description=open('README.md').read(),
    keywords = ["elasticsearch_parse","fengyun"],
    url='http://xiaorui.cc',
    author='ruifengyun',
    author_email='rfyiamcool@163.com',
    install_requires=['elasticsearch-dsl'],
    packages=['elasticsearch_parse'],
    license = "MIT",
    classifiers = [
        'Development Status :: 2 - Pre-Alpha',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 2.6',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3.0',
        'Topic :: Software Development :: Libraries :: Python Modules',
            ]
)



================================================
FILE: test.py
================================================
from elasticsearch_parse import Search, Q


s = Search(index="my-index") \
    .filter("term", blog="xiaorui.cc") \
    .query("match", author="ruifengyun")   \
    .query(~Q("match", face="good"))

s.aggs.bucket('per_tag', 'terms', field='tags')

print s.execute()



================================================
FILE: elasticsearch_parse/__init__.py
================================================
from .query import Q
from .filter import F
from .aggs import A
from .search import Search
from .index import Index
from .faceted_search import * 

VERSION = (0, 0, 9)
__version__ = VERSION
__versionstr__ = '.'.join(map(str, VERSION))



================================================
FILE: elasticsearch_parse/aggs.py
================================================
from .utils import DslBase, _make_dsl_class

__all__ = [
    'A', 'Agg', 'Filter', 'Bucket', 'Children', 'DateHistogram', 'Filters',
    'GeoDistance', 'GeohashGrid', 'Global', 'Histogram', 'Iprange', 'Missing',
    'Nested', 'Range', 'ReverseNested', 'SignificantTerms', 'Terms', 'Avg',
    'Cardinality', 'ExtendedStats', 'GeoBounds', 'Max', 'Min', 'Percentiles',
    'PercenileRanks', 'ScriptedMetric', 'Stats', 'Sum', 'TopHits', 'ValueCount'
]


def A(name_or_agg, filter=None, **params):
    if filter is not None:
        if name_or_agg != 'filter':
            raise ValueError("Aggregation %r doesn't accept positional argument 'filter'." % name_or_agg)
        params['filter'] = filter

    # {"terms": {"field": "tags"}, "aggs": {...}}
    if isinstance(name_or_agg, dict):
        if params:
            raise ValueError('A() cannot accept parameters when passing in a dict.')
        # copy to avoid modifying in-place
        agg = name_or_agg.copy()
        # pop out nested aggs
        aggs = agg.pop('aggs', None)
        # should be {"terms": {"fied": "tags"}}
        if len(agg) != 1:
            raise ValueError('A() can only accept dict with an aggregation ({"terms": {...}}). '
                 'Instead it got (%r)' % name_or_agg)
        agg_type, params = agg.popitem()
        if aggs:
            params = params.copy()
            params['aggs'] = aggs
        return Agg.get_dsl_class(agg_type)(**params)

    # Terms(...) just return the nested agg
    elif isinstance(name_or_agg, Agg):
        if params:
            raise ValueError('A() cannot accept parameters when passing in an Agg object.')
        return name_or_agg

    # "terms", field="tags"
    return Agg.get_dsl_class(name_or_agg)(**params)

class Agg(DslBase):
    _type_name = 'agg'
    _type_shortcut = staticmethod(A)
    name = None

class AggBase(object):
    _param_defs = {
        'aggs': {'type': 'agg', 'hash': True},
    }
    def __getitem__(self, agg_name):
        agg = self._params.setdefault('aggs', {})[agg_name] # propagate KeyError

        # make sure we're not mutating a shared state - whenever accessing a
        # bucket, return a shallow copy of it to be safe
        if isinstance(agg, Bucket):
            agg = A(agg.name, **agg._params)
            # be sure to store the copy so any modifications to it will affect us
            self._params['aggs'][agg_name] = agg

        return agg

    def __setitem__(self, agg_name, agg):
        self.aggs[agg_name] = A(agg)

    def _agg(self, bucket, name, agg_type, *args,**params):
        agg = self[name] = A(agg_type, *args, **params)

        # For chaining - when creating new buckets return them...
        if bucket:
            return agg
        # otherwise return self._base so we can keep chaining
        else:
            return self._base

    def metric(self, name, agg_type, *args, **params):
        return self._agg(False, name, agg_type, *args, **params)

    def bucket(self, name, agg_type, *args, **params):
        return self._agg(True, name, agg_type, *args, **params)


class Bucket(AggBase, Agg):
    def __init__(self, **params):
        super(Bucket, self).__init__(**params)
        # remember self for chaining
        self._base = self

    def to_dict(self):
        d = super(AggBase, self).to_dict()
        if 'aggs' in d[self.name]:
            d['aggs'] = d[self.name].pop('aggs')
        return d

class Filter(Bucket):
    name = 'filter'
    _param_defs = {
        'filter': {'type': 'filter'},
        'aggs': {'type': 'agg', 'hash': True},
    }

    def __init__(self, filter=None, **params):
        if filter is not None:
            params['filter'] = filter
        super(Filter, self).__init__(**params)

    def to_dict(self):
        d = super(Filter, self).to_dict()
        d[self.name].update(d[self.name].pop('filter', {}))
        return d

AGGS = (
    (Bucket, 'children', None),
    (Bucket, 'date_histogram', None),
    (Bucket, 'date_range', None),
    (Bucket, 'filters', {'filters': {'type': 'filter', 'hash': True}}),
    (Bucket, 'geo_distance', None),
    (Bucket, 'geohash_grid', None),
    (Bucket, 'global', None),
    (Bucket, 'histogram', None),
    (Bucket, 'iprange', None),
    (Bucket, 'missing', None),
    (Bucket, 'nested', None),
    (Bucket, 'range', None),
    (Bucket, 'reverse_nested', None),
    (Bucket, 'significant_terms', None),
    (Bucket, 'terms', None),

    (Agg, 'avg', None),
    (Agg, 'cardinality', None),
    (Agg, 'extended_stats', None),
    (Agg, 'geo_bounds', None),
    (Agg, 'max', None),
    (Agg, 'min', None),
    (Agg, 'percentiles', None),
    (Agg, 'percentile_ranks', None),
    (Agg, 'scripted_metric', None),
    (Agg, 'stats', None),
    (Agg, 'sum', None),
    (Agg, 'top_hits', None),
    (Agg, 'value_count', None),
)

# generate the aggregation classes dynamicaly
for base, fname, params_def in AGGS:
    # don't override the params def from AggBase
    if params_def:
        params_def.update(AggBase._param_defs)
    fclass = _make_dsl_class(base, fname, params_def)
    globals()[fclass.__name__] = fclass



================================================
FILE: elasticsearch_parse/exceptions.py
================================================
class ElasticsearchDslException(Exception):
    pass


class UnknownDslObject(ElasticsearchDslException):
    pass


class ValidationException(ValueError, ElasticsearchDslException):
    pass


class IllegalOperation(ElasticsearchDslException):
    pass



================================================
FILE: elasticsearch_parse/faceted_search.py
================================================
from datetime import timedelta, datetime
from six import iteritems, itervalues
from functools import partial

from .search import Search
from .filter import F
from .aggs import A
from .utils import AttrDict
from .result import Response

__all__ = ['FacetedSearch', 'HistogramFacet', 'TermsFacet', 'DateHistogramFacet', 'RangeFacet']

class Facet(object):
    """
    A facet on faceted search. Wraps and aggregation and provides functionality
    to create a filter for selected values and return a list of facet values
    from the result of the aggregation.
    """
    agg_type = None

    def __init__(self, **kwargs):
        self.filter_values = ()
        self._params = kwargs

    def get_aggregation(self):
        """
        Return the aggregation object.
        """
        return A(self.agg_type, **self._params)

    def add_filter(self, filter_values):
        """
        Construct a filter and remember the values for use in get_values.
        """
        self.filter_values = filter_values

        if not filter_values:
            return

        f = self.get_value_filter(filter_values[0])
        for v in filter_values[1:]:
            f |= self.get_value_filter(v)
        return f

    def get_value_filter(self, filter_value):
        """
        Construct a filter for an individual value
        """
        pass

    def is_filtered(self, key):
        """
        Is a filter active on the given key.
        """
        return key in self.filter_values

    def get_value(self, bucket):
        """
        return a value representing a bucket. Its key as default.
        """
        return bucket['key']

    def get_values(self, data):
        """
        Turn the raw bucket data into a list of tuples containing the key,
        number of documents and a flag indicating whether this value has been
        selected or not.
        """
        out = []
        for bucket in data:
            key = self.get_value(bucket)
            out.append((
                key,
                bucket['doc_count'],
                self.is_filtered(key)
            ))
        return out


class TermsFacet(Facet):
    agg_type = 'terms'

    def add_filter(self, filter_values):
        """ Create a terms filter instead of bool containing term filters.  """
        self.filter_values = filter_values
        if filter_values:
            return F('terms', **{self._params['field']: filter_values})


class RangeFacet(Facet):
    agg_type = 'range'

    def _range_to_dict(self, range):
        key, range = range
        out = {'key': key}
        if range[0] is not None:
            out['from'] = range[0]
        if range[1] is not None:
            out['to'] = range[1]
        return out

    def __init__(self, ranges, **kwargs):
        super(RangeFacet, self).__init__(**kwargs)
        self._params['ranges'] = list(map(self._range_to_dict, ranges))
        self._params['keyed'] = False
        self._ranges = dict(ranges)

    def get_value_filter(self, filter_value):
        f, t = self._ranges[filter_value]
        limits = {}
        if f is not None:
            limits['from'] = f
        if t is not None:
            limits['to'] = t

        return F('range', **{
            self._params['field']: limits
        })

class HistogramFacet(Facet):
    agg_type = 'histogram'

    def get_value_filter(self, filter_value):
        return F('range', **{
            self._params['field']: {
                'gte': filter_value,
                'lt': filter_value + self._params['interval']
            }
        })


class DateHistogramFacet(Facet):
    agg_type = 'date_histogram'

    DATE_INTERVALS = {
        'month': lambda d: (d+timedelta(days=32)).replace(day=1),
        'week': lambda d: d+timedelta(days=7),
        'day': lambda d: d+timedelta(days=1),
        'hour': lambda d: d+timedelta(hours=1),
    }

    def get_value(self, bucket):
        return datetime.utcfromtimestamp(int(bucket['key']) / 1000)

    def get_value_filter(self, filter_value):
        return F('range', **{
            self._params['field']: {
                'gte': filter_value,
                'lt': self.DATE_INTERVALS[self._params['interval']](filter_value)
            }
        })


class FacetedResponse(Response):
    def __init__(self, search, *args, **kwargs):
        super(FacetedResponse, self).__init__(*args, **kwargs)
        super(AttrDict, self).__setattr__('_search', search)

    @property
    def query_string(self):
        return self._search._query

    @property
    def facets(self):
        if not hasattr(self, '_facets'):
            super(AttrDict, self).__setattr__('_facets', AttrDict({}))
            for name, facet in iteritems(self._search.facets):
                self._facets[name] = facet.get_values(self.aggregations['_filter_' + name][name]['buckets'])
        return self._facets


class FacetedSearch(object):
    index = '_all'
    doc_types = ['_all']
    fields = ('*', )
    facets = {}

    def __init__(self, query=None, filters={}):
        self._query = query
        self._filters = {}
        for name, value in iteritems(filters):
            self.add_filter(name, value)

    def add_filter(self, name, filter_values):
        """
        Add a filter for a facet.
        """
        # normalize the value into a list
        if not isinstance(filter_values, (tuple, list)):
            if filter_values in (None, ''):
                return
            filter_values = [filter_values, ]

        # get the filter from the facet
        f = self.facets[name].add_filter(filter_values)
        if f is None:
            return

        self._filters[name] = f

    def search(self):
        """
        Construct the Search object.
        """
        return Search(doc_type=self.doc_types, index=self.index)

    def query(self, search, query):
        """
        Add query part to ``search``.

        Override this if you wish to customize the query used.
        """
        if query:
            return search.query('multi_match', fields=self.fields, query=query)
        return search

    def aggregate(self, search):
        """
        Add aggregations representing the facets selected, including potential
        filters.
        """
        for f, facet in iteritems(self.facets):
            agg = facet.get_aggregation()
            agg_filter = F('match_all')
            for field, filter in iteritems(self._filters):
                if f == field:
                    continue
                agg_filter &= filter
            search.aggs.bucket(
                '_filter_' + f,
                'filter',
                filter=agg_filter
            ).bucket(f, agg)

    def filter(self, search):
        """
        Add a ``post_filter`` to the search request narrowing the results based
        on the facet filters.
        """
        post_filter = F('match_all')
        for f in itervalues(self._filters):
            post_filter &= f
        return search.post_filter(post_filter)

    def highlight(self, search):
        """
        Add highlighting for all the fields
        """
        return search.highlight(*self.fields)

    def build_search(self):
        """
        Construct the ``Search`` object.
        """
        s = self.search()
        s = self.query(s, self._query)
        s = self.filter(s)
        s = self.highlight(s)
        self.aggregate(s)
        return s

    def execute(self):
        if not hasattr(self, '_response'):
            s = self.build_search()
            self._response = s.execute(response_class=partial(FacetedResponse, self))

        return self._response




================================================
FILE: elasticsearch_parse/filter.py
================================================
from .utils import DslBase, BoolMixin, _make_dsl_class

__all__ = [
    'F', 'And', 'AndOrFilter', 'Bool', 'EMPTY_FILTER', 'Exists', 'Filter',
    'Fquery', 'GeoBoundingBox', 'GeoDistance', 'GeoDistanceRange',
    'GeoPolygon', 'GeoShape', 'GeohashCell', 'HasChild', 'HasParent', 'Ids',
    'Indices', 'Limit', 'MatchAll', 'Missing', 'Nested', 'Not', 'Or', 'Prefix',
    'Query', 'Range', 'Regexp', 'Script', 'Term', 'Terms', 'Type'
]


def F(name_or_filter='match_all', filters=None, **params):
    # 'and/or', [F(), F()]
    if filters is not None:
        # someone passed in positional argument to F outside of and/or or query
        if name_or_filter in ('and', 'or'):
            params['filters'] = filters
        elif name_or_filter == 'query':
            params['query'] = filters
        else:
            raise ValueError("Filter %r doesn't accept positional argument." % name_or_filter)

    # {"term": {...}}
    if isinstance(name_or_filter, dict):
        if params:
            raise ValueError('F() cannot accept parameters when passing in a dict.')
        if len(name_or_filter) != 1:
            raise ValueError('F() can only accept dict with a single filter ({"bool": {...}}). '
                 'Instead it got (%r)' % name_or_filter)
        name, params = name_or_filter.copy().popitem()
        if isinstance(params, dict):
            return Filter.get_dsl_class(name)(**params)
        else:
            # and filter can have list
            return Filter.get_dsl_class(name)(params)

    # Term(...)
    if isinstance(name_or_filter, Filter):
        if params:
            raise ValueError('F() cannot accept parameters when passing in a Filter object.')
        return name_or_filter

    # s.filter = ~F(s.filter)
    if hasattr(name_or_filter, '_proxied'):
        return name_or_filter._proxied

    # 'term', tag='python', ...
    return Filter.get_dsl_class(name_or_filter)(**params)

class Filter(DslBase):
    _type_name = 'filter'
    _type_shortcut = staticmethod(F)
    name = None

class MatchAll(Filter):
    name = 'match_all'
    def __add__(self, other):
        return other._clone()
    __and__ = __rand__ = __radd__ = __add__

    def __or__(self, other):
        return self
    __ror__ = __or__
EMPTY_FILTER = MatchAll()

class Bool(BoolMixin, Filter):
    name = 'bool'
    _param_defs = {
        'must': {'type': 'filter', 'multi': True},
        'should': {'type': 'filter', 'multi': True},
        'must_not': {'type': 'filter', 'multi': True},
    }

    def __and__(self, other):
        f = self._clone()
        if isinstance(other, self.__class__):
            f.must += other.must
            f.must_not += other.must_not
            f.should = []
            if self.should and other.should:
                selfshould, othershould = self.should[:], other.should[:]
                # required subfilter, move to must
                for s in (selfshould, othershould):
                    if len(s) == 1:
                        f.must.append(s.pop())

                # we have leftover lists, nothing to do but add to must as bool(should)
                if selfshould and othershould:
                    f.must.extend((
                        Bool(should=selfshould),
                        Bool(should=othershould),
                    ))
                # at most one should list is left, keep as should
                else:
                    f.should = selfshould + othershould

            # at most one should list is left, keep as should
            else:
                f.should = self.should + other.should
        else:
            f.must.append(other)
        return f
    __rand__ = __and__

# register this as Bool for Filter
Filter._bool = Bool

class Not(Filter):
    name = 'not'
    _param_defs = {'filter': {'type': 'filter'}}

    def __init__(self, filter=None, **kwargs):
        if filter is None:
            filter, kwargs = kwargs, {}
        super(Not, self).__init__(filter=filter, **kwargs)

class AndOrFilter(object):
    _param_defs = {'filters': {'type': 'filter', 'multi': True}}

    def __init__(self, filters=None, **kwargs):
        if filters is not None:
            kwargs['filters'] = filters
        super(AndOrFilter, self).__init__(**kwargs)

    # compound filters
class And(AndOrFilter, Filter):
    name = 'and'

class Or(AndOrFilter, Filter):
    name = 'or'

class Query(Filter):
    name = 'query'
    _param_defs = {'query': {'type': 'query'}}

    def __init__(self, query=None, **kwargs):
        if query is not None:
            kwargs['query'] = query
        super(Query, self).__init__(**kwargs)

    def to_dict(self):
        d = super(Query, self).to_dict()
        d[self.name].update(d[self.name].pop('query', {}))
        return d


FILTERS = (
    # relationships
    ('nested', {'filter': {'type': 'filter'}}),
    ('has_child', {'filter': {'type': 'filter'}}),
    ('has_parent', {'filter': {'type': 'filter'}}),

    ('fquery', {'query': {'type': 'query'}}),

    # core filters
    ('exists', None),
    ('geo_bounding_box', None),
    ('geo_distance', None),
    ('geo_distance_range', None),
    ('geo_polygon', None),
    ('geo_shape', None),
    ('geohash_cell', None),
    ('ids', None),
    ('indices', None),
    ('limit', None),
    ('missing', None),
    ('prefix', None),
    ('range', None),
    ('regexp', None),
    ('script', None),
    ('term', None),
    ('terms', None),
    ('type', None),
)

# generate the filter classes dynamicaly
for fname, params_def in FILTERS:
    fclass = _make_dsl_class(Filter, fname, params_def)
    globals()[fclass.__name__] = fclass




================================================
FILE: elasticsearch_parse/index.py
================================================
from .search import Search

class Index(object):
    def __init__(self, name, using='default'):
        self._name = name
        self._doc_types = {}
        self._mappings = {}
        self._using = using
        self._settings = {}
        self._aliases = {}

    def clone(self, name, using=None):
        i = Index(name, using=using or self._using)
        for attr in ('_doc_types', '_mappings', '_settings', '_aliases'):
            setattr(i, attr, getattr(self, attr).copy())
        return i

    def _get_connection(self):
        return connections.get_connection(self._using)
    connection = property(_get_connection)

    def doc_type(self, doc_type):
        name = doc_type._doc_type.name
        self._doc_types[name] = doc_type
        self._mappings[name] = doc_type._doc_type.mapping

        if not doc_type._doc_type.index:
            doc_type._doc_type.index = self._name
        return doc_type # to use as decorator???

    def settings(self, **kwargs):
        self._settings.update(kwargs)
        return self

    def aliases(self, **kwargs):
        self._aliases.update(kwargs)
        return self

    def search(self):
        return Search(
            using=self._using,
            index=self._name,
            doc_type=[self._doc_types.get(k, k) for k in self._mappings]
        )

    def _get_mappings(self):
        analysis, mappings = {}, {}
        for mapping in self._mappings.values():
            mappings.update(mapping.to_dict())
            a = mapping._collect_analysis()
            # merge the defintion
            # TODO: conflict detection/resolution
            for key in a:
                analysis.setdefault(key, {}).update(a[key])

        return mappings, analysis

    def to_dict(self):
        out = {}
        if self._settings:
            out['settings'] = self._settings
        if self._aliases:
            out['aliases'] = self._aliases
        mappings, analysis = self._get_mappings()
        if mappings:
            out['mappings'] = mappings
        if analysis:
            out.setdefault('settings', {})['analysis'] = analysis
        return out

    def create(self, **kwargs):
        self.connection.indices.create(index=self._name, body=self.to_dict(), **kwargs)

    def delete(self, **kwargs):
        self.connection.indices.delete(index=self._name, **kwargs)



================================================
FILE: elasticsearch_parse/query.py
================================================
from .utils import DslBase, BoolMixin, _make_dsl_class
from .function import SF, ScoreFunction

__all__ = [
    'Q', 'Bool', 'Boosting', 'Common', 'ConstantScore', 'DisMax', 'Filtered',
    'FunctionScore', 'Fuzzy', 'FuzzyLikeThis', 'FuzzyLikeThisField',
    'GeoShape', 'HasChild', 'HasParent', 'Ids', 'Indices', 'Match', 'MatchAll',
    'MatchPhrase', 'MatchPhrasePrefix', 'MoreLikeThis', 'MoreLikeThisField',
    'MultiMatch', 'Nested', 'Prefix', 'Query', 'QueryString', 'Range',
    'Regexp', 'SF', 'ScoreFunction', 'SimpleQueryString', 'SpanFirst',
    'SpanMulti', 'SpanNear', 'SpanNot', 'SpanOr', 'SpanTerm', 'Template',
    'Term', 'Terms', 'TopChildren', 'Wildcard'
]


def Q(name_or_query='match_all', **params):
    # {"match": {"title": "python"}}
    if isinstance(name_or_query, dict):
        if params:
            raise ValueError('Q() cannot accept parameters when passing in a dict.')
        if len(name_or_query) != 1:
            raise ValueError('Q() can only accept dict with a single query ({"match": {...}}). '
                 'Instead it got (%r)' % name_or_query)
        name, params = name_or_query.copy().popitem()
        return Query.get_dsl_class(name)(**params)

    # MatchAll()
    if isinstance(name_or_query, Query):
        if params:
            raise ValueError('Q() cannot accept parameters when passing in a Query object.')
        return name_or_query

    # s.query = Q('filtered', query=s.query)
    if hasattr(name_or_query, '_proxied'):
        return name_or_query._proxied

    # "match", title="python"
    return Query.get_dsl_class(name_or_query)(**params)

class Query(DslBase):
    _type_name = 'query'
    _type_shortcut = staticmethod(Q)
    name = None

class MatchAll(Query):
    name = 'match_all'
    def __add__(self, other):
        return other._clone()
    __and__ = __rand__ = __radd__ = __add__

    def __or__(self, other):
        return self
    __ror__ = __or__
EMPTY_QUERY = MatchAll()

class Bool(BoolMixin, Query):
    name = 'bool'
    _param_defs = {
        'must': {'type': 'query', 'multi': True},
        'should': {'type': 'query', 'multi': True},
        'must_not': {'type': 'query', 'multi': True},
        'filter': {'type': 'query', 'multi': True},
    }

    def __and__(self, other):
        q = self._clone()
        if isinstance(other, self.__class__):
            q.must += other.must
            q.must_not += other.must_not
            q.should = []
            for qx in (self, other):
                min_should_match = getattr(qx, 'minimum_should_match', 0 if any((qx.must, qx.must_not)) else 1)
                # all subqueries are required
                if len(qx.should) <= min_should_match:
                    q.must.extend(qx.should)
                # not all of them are required, use it and remember min_should_match
                elif not q.should:
                    q.minimum_should_match = min_should_match
                    q.should = qx.should
                # not all are required, add a should list to the must with proper min_should_match
                else:
                    q.must.append(Bool(should=qx.should, minimum_should_match=min_should_match))
        else:
            q.must.append(other)
        return q
    __rand__ = __and__
# register this as Bool for Query
Query._bool = Bool

class FunctionScore(Query):
    name = 'function_score'
    _param_defs = {
        'query': {'type': 'query'},
        'filter': {'type': 'filter'},
        'functions': {'type': 'score_function', 'multi': True},
    }

    def __init__(self, **kwargs):
        if 'functions' in kwargs:
            pass
        else:
            fns = kwargs['functions'] = []
            for name in ScoreFunction._classes:
                if name in kwargs:
                    fns.append({name: kwargs.pop(name)})
        super(FunctionScore, self).__init__(**kwargs)

QUERIES = (
    # compound queries
    ('boosting', {'positive': {'type': 'query'}, 'negative': {'type': 'query'}}),
    ('constant_score', {'query': {'type': 'query'}, 'filter': {'type': 'filter'}}),
    ('dis_max', {'queries': {'type': 'query', 'multi': True}}),
    ('filtered', {'query': {'type': 'query'}, 'filter': {'type': 'filter'}}),
    ('indices', {'query': {'type': 'query'}, 'no_match_query': {'type': 'query'}}),

    # relationship queries
    ('nested', {'query': {'type': 'query'}}),
    ('has_child', {'query': {'type': 'query'}}),
    ('has_parent', {'query': {'type': 'query'}}),
    ('top_children', {'query': {'type': 'query'}}),

    # compount span queries
    ('span_first', {'match': {'type': 'query'}}),
    ('span_multi', {'match': {'type': 'query'}}),
    ('span_near', {'clauses': {'type': 'query', 'multi': True}}),
    ('span_not', {'exclude': {'type': 'query'}, 'include': {'type': 'query'}}),
    ('span_or', {'clauses': {'type': 'query', 'multi': True}}),

    # core queries
    ('common', None),
    ('fuzzy', None),
    ('fuzzy_like_this', None),
    ('fuzzy_like_this_field', None),
    ('geo_shape', None),
    ('ids', None),
    ('match', None),
    ('match_phrase', None),
    ('match_phrase_prefix', None),
    ('more_like_this', None),
    ('more_like_this_field', None),
    ('multi_match', None),
    ('prefix', None),
    ('query_string', None),
    ('range', None),
    ('regexp', None),
    ('simple_query_string', None),
    ('span_term', None),
    ('template', None),
    ('term', None),
    ('terms', None),
    ('wildcard', None),
)

# generate the query classes dynamicaly
for qname, params_def in QUERIES:
    qclass = _make_dsl_class(Query, qname, params_def)
    globals()[qclass.__name__] = qclass




================================================
FILE: elasticsearch_parse/search.py
================================================
from six import iteritems, string_types

from elasticsearch.helpers import scan

from .query import Q, EMPTY_QUERY, Filtered
from .filter import F, EMPTY_FILTER
from .aggs import A, AggBase
from .utils import DslBase

class BaseProxy(object):
    """
    Simple proxy around DSL objects (queries and filters) that can be called
    (to add query/filter) and also allows attribute access which is proxied to
    the wrapped query/filter.
    """
    def __init__(self, search, attr_name):
        self._search = search
        self._proxied = self._empty
        self._attr_name = attr_name

    def __nonzero__(self):
        return self._proxied != self._empty
    __bool__ = __nonzero__

    def __call__(self, *args, **kwargs):
        s = self._search._clone()
        getattr(s, self._attr_name)._proxied += self._shortcut(*args, **kwargs)

        # always return search to be chainable
        return s

    def __getattr__(self, attr_name):
        return getattr(self._proxied, attr_name)

    def __setattr__(self, attr_name, value):
        if not attr_name.startswith('_'):
            self._proxied = self._shortcut(self._proxied.to_dict())
            setattr(self._proxied, attr_name, value)
        super(BaseProxy, self).__setattr__(attr_name, value)


class ProxyDescriptor(object):
    """
    Simple descriptor to enable setting of queries and filters as:

        s = Search()
        s.query = Q(...)

    """
    def __init__(self, name):
        self._attr_name = '_%s_proxy' % name

    def __get__(self, instance, owner):
        return getattr(instance, self._attr_name)

    def __set__(self, instance, value):
        proxy = getattr(instance, self._attr_name)
        proxy._proxied = proxy._shortcut(value)


class ProxyQuery(BaseProxy):
    _empty = EMPTY_QUERY
    _shortcut = staticmethod(Q)


class ProxyFilter(BaseProxy):
    _empty = EMPTY_FILTER
    _shortcut = staticmethod(F)


class AggsProxy(AggBase, DslBase):
    name = 'aggs'
    def __init__(self, search):
        self._base = self._search = search
        self._params = {'aggs': {}}

    def to_dict(self):
        return super(AggsProxy, self).to_dict().get('aggs', {})


class Search(object):
    query = ProxyDescriptor('query')
    filter = ProxyDescriptor('filter')
    post_filter = ProxyDescriptor('post_filter')

    def __init__(self, using='default', index=None, doc_type=None, extra=None):
        """
        Search request to elasticsearch.

        :arg using: `Elasticsearch` instance to use
        :arg index: limit the search to index
        :arg doc_type: only query this type.

        All the paramters supplied (or omitted) at creation type can be later
        overriden by methods (`using`, `index` and `doc_type` respectively).
        """
        self._using = using

        self._index = None
        if isinstance(index, (tuple, list)):
            self._index = list(index)
        elif index:
            self._index = [index]

        self._doc_type = []
        self._doc_type_map = {}
        if isinstance(doc_type, (tuple, list)):
            for dt in doc_type:
                self._add_doc_type(dt)
        elif isinstance(doc_type, dict):
            self._doc_type.extend(doc_type.keys())
            self._doc_type_map.update(doc_type)
        elif doc_type:
            self._add_doc_type(doc_type)

        self.aggs = AggsProxy(self)
        self._sort = []
        self._extra = extra or {}
        self._params = {}
        self._fields = None
        self._partial_fields = {}
        self._highlight = {}
        self._highlight_opts = {}
        self._suggest = {}
        self._script_fields = {}

        self._query_proxy = ProxyQuery(self, 'query')
        self._filter_proxy = ProxyFilter(self, 'filter')
        self._post_filter_proxy = ProxyFilter(self, 'post_filter')

    def __getitem__(self, n):
        """
        Support slicing the `Search` instance for pagination.

        Slicing equates to the from/size parameters. E.g.::

            s = Search().query(...)[0:25]

        is equivalent to::

            s = Search().query(...).extra(from_=0, size=25)

        """
        s = self._clone()

        if isinstance(n, slice):
            # If negative slicing, abort.
            if n.start and n.start < 0 or n.stop and n.stop < 0:
                raise ValueError("Search does not support negative slicing.")
            # Elasticsearch won't get all results so we default to size: 10 if
            # stop not given.
            s._extra['from'] = n.start or 0
            s._extra['size'] = n.stop - (n.start or 0) if n.stop is not None else 10
            return s
        else:  # This is an index lookup, equivalent to slicing by [n:n+1].
            # If negative index, abort.
            if n < 0:
                raise ValueError("Search does not support negative indexing.")
            s._extra['from'] = n
            s._extra['size'] = 1
            return s

    @classmethod
    def from_dict(cls, d):
        """
        Construct a `Search` instance from a raw dict containing the search
        body. Useful when migrating from raw dictionaries.

        Example::

            s = Search.from_dict({
                "query": {
                    "bool": {
                        "must": [...]
                    }
                },
                "aggs": {...}
            })
            s = s.filter('term', published=True)
        """
        s = cls()
        s.update_from_dict(d)
        return s

    def _clone(self):
        """
        Return a clone of the current search request. Performs a shallow copy
        of all the underlying objects. Used internally by most state modifying
        APIs.
        """
        s = self.__class__(using=self._using, index=self._index,
                           doc_type=self._doc_type)
        s._doc_type_map = self._doc_type_map.copy()
        s._sort = self._sort[:]
        s._fields = self._fields[:] if self._fields is not None else None
        s._partial_fields = self._partial_fields.copy()
        s._extra = self._extra.copy()
        s._highlight = self._highlight.copy()
        s._highlight_opts = self._highlight_opts.copy()
        s._suggest = self._suggest.copy()
        s._script_fields = self._script_fields.copy()
        for x in ('query', 'filter', 'post_filter'):
            getattr(s, x)._proxied = getattr(self, x)._proxied

        # copy top-level bucket definitions
        if self.aggs._params.get('aggs'):
            s.aggs._params = {'aggs': self.aggs._params['aggs'].copy()}
        s._params = self._params.copy()
        return s

    def update_from_dict(self, d):
        """
        Apply options from a serialized body to the current instance. Modifies
        the object in-place. Used mostly by ``from_dict``.
        """
        d = d.copy()
        if 'query' in d:
            self.query._proxied = Q(d.pop('query'))
        if 'post_filter' in d:
            self.post_filter._proxied = F(d.pop('post_filter'))

        if isinstance(self.query._proxied, Filtered):
            self.filter._proxied = self.query._proxied.filter
            self.query._proxied = self.query._proxied.query

        aggs = d.pop('aggs', d.pop('aggregations', {}))
        if aggs:
            self.aggs._params = {
                'aggs': dict(
                    (name, A(value)) for (name, value) in iteritems(aggs))
            }
        if 'sort' in d:
            self._sort = d.pop('sort')
        if 'fields' in d:
            self._fields = d.pop('fields')
        if 'partial_fields' in d:
            self._partial_fields = d.pop('partial_fields')
        if 'highlight' in d:
            high = d.pop('highlight').copy()
            self._highlight = high.pop('fields')
            self._highlight_opts = high
        if 'suggest' in d:
            self._suggest = d.pop('suggest')
            if 'text' in self._suggest:
                text = self._suggest.pop('text')
                for s in self._suggest.values():
                    s.setdefault('text', text)
        if 'script_fields' in d:
            self._script_fields = d.pop('script_fields')
        self._extra = d

    def script_fields(self, **kwargs):
        """
        Define script fields to be calculated on hits. See
        https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html
        for more details.

        Example::

            s = Search()
            s = s.script_fields(times_two="doc['field'].value * 2")
            s = s.script_fields(
                times_three={
                    'script': "doc['field'].value * n",
                    'params': {'n': 3}
                }
            )
        
        """
        s = self._clone()
        for name in kwargs:
            if isinstance(kwargs[name], string_types):
                kwargs[name] = {'script': kwargs[name]}
        s._script_fields.update(kwargs)
        return s

    def params(self, **kwargs):
        """
        Specify query params to be used when executing the search. All the
        keyword arguments will override the current values. See
        http://elasticsearch-py.readthedocs.org/en/master/api.html#elasticsearch.Elasticsearch.search
        for all availible parameters.

        Example::

            s = Search()
            s = s.params(routing='user-1', preference='local')
        """
        s = self._clone()
        s._params.update(kwargs)
        return s

    def extra(self, **kwargs):
        """
        Add extra keys to the request body. Mostly here for backwards
        compatibility.
        """
        s = self._clone()
        if 'from_' in kwargs:
            kwargs['from'] = kwargs.pop('from_')
        s._extra.update(kwargs)
        return s

    def fields(self, fields=None):
        """
        Selectively load specific stored fields for each document.

        :arg fields: list of fields to return for each document

        If ``fields`` is None, the entire document will be returned for
        each hit.  If fields is the empty list, no fields will be
        returned for each hit, just the metadata.
        """
        s = self._clone()
        s._fields = fields
        return s

    def partial_fields(self, **partial):
        """
        Control which part of the fields to extract from the `_source` document

        :kwargs partial: dict specifying which fields to extract from the source

        An example usage would be:

            s = Search().partial_fields(authors_data={
                    'include': ['authors.*'],
                    'exclude': ['authors.name']
                })

        which will include all fields from the `authors` nested property except for
        each authors `name`

        If ``partial`` is not provided, the whole `_source` will be fetched. Calling this multiple
        times will override the previous values with the new ones.
        """
        s = self._clone()
        s._partial_fields = partial
        return s

    def sort(self, *keys):
        """
        Add sorting information to the search request. If called without
        arguments it will remove all sort requirements. Otherwise it will
        replace them. Acceptable arguments are::

            'some.field'
            '-some.other.field'
            {'different.field': {'any': 'dict'}}

        so for example::

            s = Search().sort(
                'category',
                '-title',
                {"price" : {"order" : "asc", "mode" : "avg"}}
            )

        will sort by ``category``, ``title`` (in descending order) and
        ``price`` in ascending order using the ``avg`` mode.

        The API returns a copy of the Search object and can thus be chained.
        """
        s = self._clone()
        s._sort = []
        for k in keys:
            if isinstance(k, string_types) and k.startswith('-'):
                k = {k[1:]: {"order": "desc"}}
            s._sort.append(k)
        return s

    def highlight_options(self, **kwargs):
        """
        Update the global highlighting options used for this request. For
        example::

            s = Search()
            s = s.highlight_options(order='score')
        """
        s = self._clone()
        s._highlight_opts.update(kwargs)
        return s

    def highlight(self, *fields, **kwargs):
        """
        Request highliting of some fields. All keyword arguments passed in will be
        used as parameters. Example::

            Search().highlight('title', 'body', fragment_size=50)

        will produce the equivalent of::

            {
                "highlight": {
                    "fields": {
                        "body": {"fragment_size": 50},
                        "title": {"fragment_size": 50}
                    }
                }
            }

        """
        s = self._clone()
        for f in fields:
            s._highlight[f] = kwargs
        return s

    def suggest(self, name, text, **kwargs):
        """
        Add a suggestions request to the search.

        :arg name: name of the suggestion
        :arg text: text to suggest on

        All keyword arguments will be added to the suggestions body. For example::

            s = Search()
            s = s.suggest('suggestion-1', 'Elasticserach', term={'field': 'body'})
        """
        s = self._clone()
        s._suggest[name] = {'text': text}
        s._suggest[name].update(kwargs)
        return s

    def index(self, *index):
        """
        Set the index for the search. If called empty it will rmove all information.

        Example:

            s = Search()
            s = s.index('twitter-2015.01.01', 'twitter-2015.01.02')
        """
        # .index() resets
        s = self._clone()
        if not index:
            s._index = None
        else:
            s._index = (self._index or []) + list(index)
        return s

    def _add_doc_type(self, doc_type):
        if hasattr(doc_type, '_doc_type'):
            self._doc_type_map[doc_type._doc_type.name] = doc_type.from_es
            doc_type = doc_type._doc_type.name
        self._doc_type.append(doc_type)

    def doc_type(self, *doc_type, **kwargs):
        """
        Set the type to search through. You can supply a single value or
        multiple. Values can be strings or subclasses of ``DocType``.
        
        You can also pass in any keyword arguments, mapping a doc_type to a
        callback that should be used instead of the Result class.

        If no doc_type is supplied any information stored on the instance will
        be erased.

        Example:

            s = Search().doc_type('product', 'store', User, custom=my_callback)
        """
        # .doc_type() resets
        s = self._clone()
        if not doc_type and not kwargs:
            s._doc_type = []
            s._doc_type_map = {}
        else:
            for dt in doc_type:
                s._add_doc_type(dt)
            s._doc_type.extend(kwargs.keys())
            s._doc_type_map.update(kwargs)
        return s

    def to_dict(self, count=False, **kwargs):
        """
        Serialize the search into the dictionary that will be sent over as the
        request's body.

        :arg count: a flag to specify we are interested in a body for count -
            no aggregations, no pagination bounds etc.

        All additional keyword arguments will be included into the dictionary.
        """
        if self.filter:
            d = {
              "query": {
                "filtered": {
                  "query": self.query.to_dict(),
                  "filter": self.filter.to_dict()
                }
              }
            }
        else:
            d = {"query": self.query.to_dict()}

        if self.post_filter:
            d['post_filter'] = self.post_filter.to_dict()

        # count request doesn't care for sorting and other things
        if not count:
            if self.aggs.aggs:
                d.update(self.aggs.to_dict())

            if self._sort:
                d['sort'] = self._sort

            d.update(self._extra)

            if self._fields is not None:
                d['fields'] = self._fields

            if self._partial_fields:
                d['partial_fields'] = self._partial_fields

            if self._highlight:
                d['highlight'] = {'fields': self._highlight}
                d['highlight'].update(self._highlight_opts)

            if self._suggest:
                d['suggest'] = self._suggest

            if self._script_fields:
                d['script_fields'] = self._script_fields

        d.update(kwargs)
        return d

    def using(self, client):
        """
        Associate the search request with an elasticsearch client. A fresh copy
        will be returned with current instance remaining unchanged.

        :arg client: an instance of ``elasticsearch.Elasticsearch`` to use or
            an alias to look up in ``elasticsearch_dsl.connections``

        """
        s = self._clone()
        s._using = client
        return s

    def count(self):
        """
        Return the number of hits matching the query and filters. Note that
        only the actual number is returned.
        """
        es = connections.get_connection(self._using)

        d = self.to_dict(count=True)
        return d

    def execute(self):
        """
        Execute the search and return an instance of ``Response`` wrapping all
        the data.

        :arg response_class: optional subclass of ``Response`` to use instead.
        """
        return self.to_dict()
        #return response_class(
        #    es.search(
        #        index=self._index,
        #        doc_type=self._doc_type,
        #        body=self.to_dict(),
        #        **self._params
        #    ),
        #    callbacks=self._doc_type_map
        #)

    def scan(self):
        """
        Turn the search into a scan search and return a generator that will
        iterate over all the documents matching the query.

        Use ``params`` method to specify any additional arguments you with to
        pass to the underlying ``scan`` helper from ``elasticsearch-py`` -
        http://elasticsearch-py.readthedocs.org/en/master/helpers.html#elasticsearch.helpers.scan

        """
        es = connections.get_connection(self._using)

        for hit in scan(
                es,
                query=self.to_dict(),
                index=self._index,
                doc_type=self._doc_type,
                **self._params
            ):
            yield self._doc_type_map.get(hit['_type'], Result)(hit)




================================================
FILE: elasticsearch_parse/serializer.py
================================================
from elasticsearch.serializer import JSONSerializer

from .utils import AttrDict, AttrList

class AttrJSONSerializer(JSONSerializer):
    def default(self, data):
        if isinstance(data, AttrDict):
            return data._d_
        if isinstance(data, AttrList):
            return data._l_
        return super(AttrJSONSerializer, self).default(data)

serializer = AttrJSONSerializer()



================================================
FILE: elasticsearch_parse/utils.py
================================================
from __future__ import unicode_literals

from six import iteritems, add_metaclass
from six.moves import map

from .exceptions import UnknownDslObject, ValidationException

SKIP_VALUES = ('', None)

def _wrap(val, obj_wrapper=None):
    if isinstance(val, dict):
        return AttrDict(val) if obj_wrapper is None else obj_wrapper(val)
    if isinstance(val, list):
        return AttrList(val)
    return val

def _make_dsl_class(base, name, params_def=None, suffix=''):
    """
    Generate a DSL class based on the name of the DSL object and it's parameters
    """
    attrs = {'name': name}
    if params_def:
        attrs['_param_defs'] = params_def
    cls_name = str(''.join(s.title() for s in name.split('_')) + suffix)
    return type(cls_name, (base, ), attrs)

class AttrList(object):
    def __init__(self, l, obj_wrapper=None):
        # make iteables into lists
        if not isinstance(l, list):
            l = list(l)
        self._l_ = l
        self._obj_wrapper = obj_wrapper

    def __repr__(self):
        return repr(self._l_)

    def __eq__(self, other):
        if isinstance(other, AttrList):
            return other._l_ == self._l_
        # make sure we still equal to a dict with the same data
        return other == self._l_

    def __getitem__(self, k):
        l = self._l_[k]
        if isinstance(k, slice):
            return AttrList(l)
        return _wrap(l, self._obj_wrapper)

    def __setitem__(self, k, value):
        self._l_[k] = value

    def __iter__(self):
        return map(lambda i: _wrap(i, self._obj_wrapper), self._l_)

    def __len__(self):
        return len(self._l_)

    def __nonzero__(self):
        return bool(self._l_)
    __bool__ = __nonzero__

    def __getattr__(self, name):
        return getattr(self._l_, name)


class AttrDict(object):
    """
    Helper class to provide attribute like access (read and write) to
    dictionaries. Used to provide a convenient way to access both results and
    nested dsl dicts.
    """
    def __init__(self, d):
        # assign the inner dict manually to prevent __setattr__ from firing
        super(AttrDict, self).__setattr__('_d_', d)

    def __contains__(self, key):
        return key in self._d_

    def __nonzero__(self):
        return bool(self._d_)
    __bool__ = __nonzero__

    def __dir__(self):
        # introspection for auto-complete in IPython etc
        return list(self._d_.keys())

    def __eq__(self, other):
        if isinstance(other, AttrDict):
            return other._d_ == self._d_
        # make sure we still equal to a dict with the same data
        return other == self._d_

    def __repr__(self):
        r = repr(self._d_)
        if len(r) > 60:
            r = r[:60] + '...}'
        return r

    def __getattr__(self, attr_name):
        try:
            return _wrap(self._d_[attr_name])
        except KeyError:
            raise AttributeError(
                '%r object has no attribute %r' % (self.__class__.__name__, attr_name))

    def __delattr__(self, attr_name):
        try:
            del self._d_[attr_name]
        except KeyError:
            raise AttributeError(
                '%r object has no attribute %r' % (self.__class__.__name__, attr_name))

    def __getitem__(self, key):
        return _wrap(self._d_[key])

    def __setitem__(self, key, value):
        self._d_[key] = value

    def __delitem__(self, key):
        del self._d_[key]

    def __setattr__(self, name, value):
        if name in self._d_ or not hasattr(self.__class__, name):
            self._d_[name] = value
        else:
            # there is an attribute on the class (could be property, ..) - don't add it as field
            super(AttrDict, self).__setattr__(name, value)

    def __iter__(self):
        return iter(self._d_)

    def to_dict(self):
        return self._d_


class DslMeta(type):
    """
    Base Metaclass for DslBase subclasses that builds a registry of all classes
    for given DslBase subclass (== all the query types for the Query subclass
    of DslBase).

    It then uses the information from that registry (as well as `name` and
    `shortcut` attributes from the base class) to construct any subclass based
    on it's name.

    For typical use see `QueryMeta` and `Query` in `elasticsearch_dsl.query`.
    """
    _types = {}
    def __init__(cls, name, bases, attrs):
        super(DslMeta, cls).__init__(name, bases, attrs)
        # skip for DslBase
        if not hasattr(cls, '_type_shortcut'):
            return
        if cls.name is None:
            # abstract base class, register it's shortcut
            cls._types[cls._type_name] = cls._type_shortcut
            # and create a registry for subclasses
            if not hasattr(cls, '_classes'):
                cls._classes = {}
        elif cls.name not in cls._classes:
            # normal class, register it
            cls._classes[cls.name] = cls

    @classmethod
    def get_dsl_type(cls, name):
        try:
            return cls._types[name]
        except KeyError:
            raise UnknownDslObject('DSL type %s does not exist.' % name)


@add_metaclass(DslMeta)
class DslBase(object):
    """
    Base class for all DSL objects - queries, filters, aggregations etc. Wraps
    a dictionary representing the object's json.

    Provides several feature:
        - attribute access to the wrapped dictionary (.field instead of ['field'])
        - _clone method returning a deep copy of self
        - to_dict method to serialize into dict (to be sent via elasticsearch-py)
        - basic logical operators (&, | and ~) using a Bool(Filter|Query) TODO:
          move into a class specific for Query/Filter
        - respects the definiton of the class and (de)serializes it's
          attributes based on the `_param_defs` definition (for example turning
          all values in the `must` attribute into Query objects)
    """
    _param_defs = {}

    @classmethod
    def get_dsl_class(cls, name):
        try:
            return cls._classes[name]
        except KeyError:
            raise UnknownDslObject('DSL class `%s` does not exist in %s.' % (name, cls._type_name))

    def __init__(self, **params):
        self._params = {}
        for pname, pvalue in iteritems(params):
            if '__' in pname:
                pname = pname.replace('__', '.')
            self._setattr(pname, pvalue)

    def _repr_params(self):
        """ Produce a repr of all our parameters to be used in __repr__. """
        return  ', '.join(
            '%s=%r' % (n.replace('.', '__'), v)
            for (n, v) in sorted(iteritems(self._params))
            # make sure we don't include empty typed params
            if 'type' not in self._param_defs.get(n, {}) or v
        )

    def __repr__(self):
        return '%s(%s)' % (
            self.__class__.__name__,
            self._repr_params()
        )

    def __eq__(self, other):
        return isinstance(other, self.__class__) and other.to_dict() == self.to_dict()

    def __ne__(self, other):
        return not self == other

    def __setattr__(self, name, value):
        if name.startswith('_'):
            return super(DslBase, self).__setattr__(name, value)
        return self._setattr(name, value)

    def _setattr(self, name, value):
        # if this attribute has special type assigned to it...
        if name in self._param_defs:
            pinfo = self._param_defs[name]

            if 'type' in pinfo:
                # get the shortcut used to construct this type (query.Q, aggs.A, etc)
                shortcut = self.__class__.get_dsl_type(pinfo['type'])
                if pinfo.get('multi'):
                    value = list(map(shortcut, value))

                # dict(name -> DslBase), make sure we pickup all the objs
                elif pinfo.get('hash'):
                    value = dict((k, shortcut(v)) for (k, v) in iteritems(value))

                # single value object, just convert
                else:
                    value = shortcut(value)
        self._params[name] = value

    def __getattr__(self, name):
        if name.startswith('_'):
            raise AttributeError(
                '%r object has no attribute %r' % (self.__class__.__name__, name))

        value = None
        try:
            value = self._params[name]
        except KeyError:
            # compound types should never throw AttributeError and return empty
            # container instead
            if name in self._param_defs:
                pinfo = self._param_defs[name]
                if pinfo.get('multi'):
                    value = self._params.setdefault(name, [])
                elif pinfo.get('hash'):
                    value = self._params.setdefault(name, {})
        if value is None:
            raise AttributeError(
                '%r object has no attribute %r' % (self.__class__.__name__, name))

        # wrap nested dicts in AttrDict for convenient access
        if isinstance(value, dict):
            return AttrDict(value)
        return value

    def to_dict(self):
        """
        Serialize the DSL object to plain dict
        """
        d = {}
        for pname, value in iteritems(self._params):
            pinfo = self._param_defs.get(pname)

            # typed param
            if pinfo and 'type' in pinfo:
                # don't serialize empty lists and dicts for typed fields
                if value in ({}, []):
                    continue

                # multi-values are serialized as list of dicts
                if pinfo.get('multi'):
                    value = list(map(lambda x: x.to_dict(), value))

                # squash all the hash values into one dict
                elif pinfo.get('hash'):
                    value = dict((k, v.to_dict()) for k, v in iteritems(value))

                # serialize single values
                else:
                    value = value.to_dict()

            # serialize anything with to_dict method
            elif hasattr(value, 'to_dict'):
                value = value.to_dict()

            d[pname] = value
        return {self.name: d}

    def _clone(self):
        return self._type_shortcut(self.to_dict())

    def __add__(self, other):
        # make sure we give queries that know how to combine themselves
        # preference
        if hasattr(other, '__radd__'):
            return other.__radd__(self)
        return self._bool(must=[self, other])

    def __invert__(self):
        return self._bool(must_not=[self])

    def __or__(self, other):
        # make sure we give queries that know how to combine themselves
        # preference
        if hasattr(other, '__ror__'):
            return other.__ror__(self)
        return self._bool(should=[self, other])

    def __and__(self, other):
        # make sure we give queries that know how to combine themselves
        # preference
        if hasattr(other, '__rand__'):
            return other.__rand__(self)
        return self._bool(must=[self, other])


class BoolMixin(object):
    """
    Mixin containing all the operator overrides for Bool queries and filters.

    Except for and where should behavior differs
    """
    def __add__(self, other):
        q = self._clone()
        if isinstance(other, self.__class__):
            q.must += other.must
            q.should += other.should
            q.must_not += other.must_not
        else:
            q.must.append(other)
        return q
    __radd__ = __add__

    def __or__(self, other):
        if not (self.must or self.must_not):
            # TODO: if only 1 in must or should, append the query instead of other
            q = self._clone()
            q.should.append(other)
            return q

        elif isinstance(other, self.__class__) and not (other.must or other.must_not):
            # TODO: if only 1 in must or should, append the query instead of self
            q = other._clone()
            q.should.append(self)
            return q

        return self.__class__(should=[self, other])
    __ror__ = __or__

    def __invert__(self):
        # special case for single negated query
        if not (self.must or self.should) and len(self.must_not) == 1:
            return self.must_not[0]._clone()

        # bol without should, just flip must and must_not
        elif not self.should:
            q = self._clone()
            q.must, q.must_not = q.must_not, q.must
            return q

        # TODO: should -> must_not.append(self.__class__(should=self.should)) ??
        # queries with should just invert normally
        return super(BoolMixin, self).__invert__()


class ObjectBase(AttrDict):
    def __init__(self, **kwargs):
        m = self._doc_type.mapping
        for k in m:
            if k in kwargs and m[k]._coerce:
                kwargs[k] = m[k].to_python(kwargs[k])
        super(ObjectBase, self).__init__(kwargs)

    def __getattr__(self, name):
        try:
            return super(ObjectBase, self).__getattr__(name)
        except AttributeError:
            if name in self._doc_type.mapping:
                f = self._doc_type.mapping[name]
                if hasattr(f, 'empty'):
                    value = f.empty()
                    if value not in SKIP_VALUES:
                        setattr(self, name, value)
                        value = getattr(self, name)
                    return value
            raise

    def __setattr__(self, name, value):
        if name in self._doc_type.mapping:
            value = self._doc_type.mapping[name].to_python(value)
        super(ObjectBase, self).__setattr__(name, value)

    def to_dict(self):
        out = {}
        for k, v in iteritems(self._d_):
            if isinstance(v, (AttrList, list, tuple)):
                v = [i.to_dict() if hasattr(i, 'to_dict') else i for i in v]
            else:
                v = v.to_dict() if hasattr(v, 'to_dict') else v

            # don't serialize empty values
            # careful not to include numeric zeros
            if v in ([], {}, None):
                continue

            out[k] = v
        return out

    def clean_fields(self):
        errors = {}
        for name in self._doc_type.mapping:
            field = self._doc_type.mapping[name]
            data = self._d_.get(name, None)
            try:
                # save the cleaned value
                self._d_[name] = field.clean(data)
            except ValidationException as e:
                errors.setdefault(name, []).append(e)

        if errors:
            raise ValidationException(errors)

    def clean(self):
        pass

    def full_clean(self):
        self.clean_fields()
        self.clean()

def merge(data, new_data):
    if not (isinstance(data, (AttrDict, dict))
            and isinstance(new_data, (AttrDict, dict))):
        raise ValueError('You can only merge two dicts! Got %r and %r instead.' % (data, new_data))

    for key, value in iteritems(new_data):
        if key in data and isinstance(data[key], (AttrDict, dict)):
            merge(data[key], value)
        else:
            data[key] = value