Welcome, guest | Sign In | My Account | Store | Cart

Notice! PyPM is being replaced with the ActiveState Platform, which enhances PyPM’s build and deploy capabilities. Create your free Platform account to download ActivePython or customize Python with the packages you require and get automatic updates.

Download
ActivePython
INSTALL>
pypm install rawes

How to install rawes

  1. Download and install ActivePython
  2. Open Command Prompt
  3. Type pypm install rawes
 Python 2.7Python 3.2Python 3.3
Windows (32-bit)
Windows (64-bit)
Mac OS X (10.5+)
0.5.3 Available View build log
Linux (32-bit)
0.5.3 Available View build log
0.3.4 Available View build log
0.2 Available View build log
Linux (64-bit)
0.5.3 Available View build log
0.4.0 Available View build log
0.2 Available View build log
 
Author
License
Apache-2.0
Lastest release
version 0.5.3 on Jul 8th, 2013

rawes

About

rawes is an elasticsearch driver for Python. It provides a small level of abstraction above the requests library - enough abstraction to be useful, but not so much to obscure elasticsearch's great [native api](http://www.elasticsearch.org/guide/reference/api/)

Features
  • elasticsearch native API support
  • Python 3 support
  • gzip over HTTP support
  • HTTPS support
  • Thrift support
Installation

`bash $ pip install rawes `

Usage

Create a connection to elasticsearch `python import rawes es = rawes.Elastic('localhost:9200') `

Search for a document ```python es.get('tweets/tweet/_search', data={

System Message: WARNING/2 (<string>, line 30); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 30); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 33)

Unexpected indentation.
'query' : {
'match_all' : {}

System Message: WARNING/2 (<string>, line 35)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 36)

Block quote ends without a blank line; unexpected unindent.
})

The rawes.Elastic constructor takes the following parameters (defaults shown): ```python rawes.Elastic(

System Message: WARNING/2 (<string>, line 39); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 39); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 42)

Unexpected indentation.
url='http://localhost:9200', # Protocol, host, and port of elasticsearch service.
# Valid protocols: http, https, thrift # Default protocol is http, unless port is in range 9500-9600, then thrift # Default ports: http=9200, https=443, thrift=9500

System Message: WARNING/2 (<string>, line 46)

Definition list ends without a blank line; unexpected unindent.

timeout=30, # Timeout in seconds, **kwrgs # http(s) only: additional parameters you wish to pass

System Message: WARNING/2 (<string>, line 46); backlink

Inline strong start-string without end-string.

System Message: ERROR/3 (<string>, line 48)

Unexpected indentation.
# to the python 'requests' library (for example, basic auth)

System Message: WARNING/2 (<string>, line 49)

Block quote ends without a blank line; unexpected unindent.
)

Constructor examples:

`python es = rawes.Elastic()                        # will connect to: http://localhost:9200 es = rawes.Elastic('https://localhost')     # will connect to: https://localhost:443 es = rawes.Elastic('thrift://localhost')    # will connect to: thrift://localhost:9500 es = rawes.Elastic('https://example.org:8443', auth=('user','pass'))  # https with basic auth connection to: https://example.org:8443 `

An instance of rawes.Elastic ('es' in this case) has methods for get, post, put, delete, and head (for each http verb). Each method takes the following parameters (defaults shown): ```python es.get(

System Message: WARNING/2 (<string>, line 61); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 61); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 64)

Unexpected indentation.

path='', # http URL path data='', # http body. can be either a string or a python dictionary (will automatically be converted to JSON) params={}, # http URL params passed as a python dictionary headers={}, # http headers as a python dictionary **kwargs # additional parameters you wish to pass to the python 'requests' library or the thrift RestRequest

System Message: WARNING/2 (<string>, line 64); backlink

Inline strong start-string without end-string.

System Message: ERROR/3 (<string>, line 69)

Unexpected indentation.
# Examples: headers, basic auth

System Message: WARNING/2 (<string>, line 70)

Block quote ends without a blank line; unexpected unindent.
)
Examples

Create a new document in the twitter index of type tweet with id 1 ```python es.put('tweets/tweet/1', data={

System Message: WARNING/2 (<string>, line 76); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 76); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 79)

Unexpected indentation.
'user' : 'dwnoble', 'post_date' : '2012-8-27T08:00:30Z', 'message' : 'Tweeting about elasticsearch'

System Message: WARNING/2 (<string>, line 82)

Block quote ends without a blank line; unexpected unindent.

}) es.put('blogs/post/2', data={

System Message: ERROR/3 (<string>, line 84)

Unexpected indentation.
'user' : 'dan', 'post_date' : '2012-8-27T09:30:03Z', 'title' : 'Elasticsearch', 'body' : 'Blogging about elasticsearch'

System Message: WARNING/2 (<string>, line 88)

Block quote ends without a blank line; unexpected unindent.
})

Search for a document, specifying http params ```python es.get('tweets/tweet/_search', data={

System Message: WARNING/2 (<string>, line 91); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 91); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 94)

Unexpected indentation.
'query' : {
'match_all' : {}

System Message: WARNING/2 (<string>, line 96)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 97)

Block quote ends without a blank line; unexpected unindent.
}, params= {
'size': 2

System Message: WARNING/2 (<string>, line 99)

Definition list ends without a blank line; unexpected unindent.
})

Search for a document with a JSON string ```python es.get('tweets,blogs/_search', data=""" {

System Message: WARNING/2 (<string>, line 102); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 102); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 106)

Unexpected indentation.
"query" : {
"match_all" : {}

System Message: WARNING/2 (<string>, line 108)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 109)

Block quote ends without a blank line; unexpected unindent.

} """) ```

System Message: WARNING/2 (<string>, line 109); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 109); backlink

Inline interpreted text or phrase reference start-string without end-string.

Update a document ```python es.put('someindex/sometype/123', data={

System Message: WARNING/2 (<string>, line 113); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 113); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 116)

Unexpected indentation.
'value' : 100, 'other' : 'stuff'

System Message: WARNING/2 (<string>, line 118)

Block quote ends without a blank line; unexpected unindent.

}) es.post('someindex/sometype/123/_update', data={

System Message: ERROR/3 (<string>, line 120)

Unexpected indentation.

'script' : 'ctx._source.value += value', 'params' : {

System Message: ERROR/3 (<string>, line 122)

Unexpected indentation.
'value' : 50

System Message: WARNING/2 (<string>, line 123)

Block quote ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 124)

Block quote ends without a blank line; unexpected unindent.
})

Delete a document `python es.delete('tweets/tweet/1') `

Bulk load ```python bulk_body = ''' {"index" : {}} {"key":"value1"} {"index" : {}} {"key":"value2"} {"index" : {}} {"key":"value3"} '''

System Message: WARNING/2 (<string>, line 132); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 132); backlink

Inline interpreted text or phrase reference start-string without end-string.

es.post('someindex/sometype/_bulk', data=bulk_body)

bulk_list = [
{"index" : {}}, {"key":"value4"}, {"index" : {}}, {"key":"value5"}, {"index" : {}}, {"key":"value6"}

System Message: WARNING/2 (<string>, line 152)

Definition list ends without a blank line; unexpected unindent.

]

# Remember to include the trailing n character for bulk inserts bulk_body_2 = 'n'.join(map(json.dumps, bulk_list))+'n'

es.post('someindex/sometype/_bulk', data=bulk_body_2) ```

System Message: WARNING/2 (<string>, line 157); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 157); backlink

Inline interpreted text or phrase reference start-string without end-string.
Alternate Syntax

Instead of setting the first argument of a es.&lt;http verb&gt; call to the HTTP URL path, you can also use python attributes and item accessors to build up the url path. For example: ```python es.post('tweets/tweet/', data={

System Message: WARNING/2 (<string>, line 163); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 163); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 166)

Unexpected indentation.
'user' : 'dwnoble', 'post_date' : '2012-8-27T09:15:59', 'message' : 'More tweets about elasticsearch'

System Message: WARNING/2 (<string>, line 169)

Block quote ends without a blank line; unexpected unindent.
})

Becomes: ```python es.tweets.tweet.post(data={

System Message: WARNING/2 (<string>, line 172); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 172); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 175)

Unexpected indentation.
'user' : 'dwnoble', 'post_date' : '2012-8-27T09:15:59', 'message' : 'More tweets about elasticsearch'

System Message: WARNING/2 (<string>, line 178)

Block quote ends without a blank line; unexpected unindent.
})

Or using item accessors ([] notation). This can be useful for characters that are not allowed in python attributes: ```python es['tweets']['tweet'].post(data={

System Message: WARNING/2 (<string>, line 181); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 181); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 184)

Unexpected indentation.
'user' : 'dwnoble', 'post_date' : '2012-8-27T09:15:59', 'message' : 'More tweets about elasticsearch'

System Message: WARNING/2 (<string>, line 187)

Block quote ends without a blank line; unexpected unindent.
})

More examples:

Searching the "tweets" index for documents of type "tweets" `python es.tweets.tweet._search.get(data={'query' : {'match_all' : {} }}) `

Searching the "tweets" and "blogs" index for documents of any type using a JSON strings `python es['tweets,blogs']._search.get(data='{"query" : {"match_all" : {}}}') `

JSON Encoding

By default, rawes will encode datetimes (timezone required!) to UTC ISO8601 strings with 'second' precision before handing the JSON off to elasticsearch. If elasticsearch has no mapping defined, this will result in the default mapping of 'dateOptionalTime.' Timezones are required for this automatic serialization: you may want to use a python module like python-dateutil (Python 2.x only) or pytz to make your life easier. ```python from datetime import datetime from dateutil import tz eastern_timezone = tz.gettz('America/New_York')

System Message: WARNING/2 (<string>, line 205); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 205); backlink

Inline interpreted text or phrase reference start-string without end-string.
es.put('tweets/tweet/99', data={
'user' : 'dwnoble', 'post_date' : datetime(2012, 8, 27, 8, 0, 30, tzinfo=eastern_timezone), 'message' : 'Tweeting about elasticsearch'

System Message: WARNING/2 (<string>, line 216)

Definition list ends without a blank line; unexpected unindent.

})

es.get('tweets/tweet/99')['_source']['post_date'] # Returns: u'2012-08-27T12:00:30Z' ```

System Message: WARNING/2 (<string>, line 218); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 218); backlink

Inline interpreted text or phrase reference start-string without end-string.

Alternatively, you can specify a custom JSON encoder using the json_encoder parameter: ```python from datetime import datetime from dateutil import tz eastern_timezone = tz.gettz('America/New_York')

System Message: WARNING/2 (<string>, line 223); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 223); backlink

Inline interpreted text or phrase reference start-string without end-string.
def encode_custom(obj):
if isinstance(obj, datetime):
return obj.astimezone(tz.tzutc()).strftime('%Y-%m-%d')

System Message: WARNING/2 (<string>, line 232)

Definition list ends without a blank line; unexpected unindent.

raise TypeError(repr(obj) + " is not JSON serializable")

es.put('tweets/tweet/445', data={
'user' : 'dwnoble', 'post_date' : datetime(2012, 11, 12, 9, 45, 45, tzinfo=eastern_timezone), 'message' : 'Tweeting about elasticsearch'

System Message: WARNING/2 (<string>, line 238)

Definition list ends without a blank line; unexpected unindent.

}, json_encoder=encode_custom)

es.get('tweets/tweet/445')['_source']['post_date'] # Returns: u'2012-11-12' ```

System Message: WARNING/2 (<string>, line 240); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 240); backlink

Inline interpreted text or phrase reference start-string without end-string.

Additionally, a default JSON encoder can be specified in the rawes.Elastic constructor:

```python import rawes from datetime import datetime from dateutil import tz eastern_timezone = tz.gettz('America/New_York')

System Message: WARNING/2 (<string>, line 247); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 247); backlink

Inline interpreted text or phrase reference start-string without end-string.
def encode_custom(obj):
if isinstance(obj, datetime):
return obj.astimezone(tz.tzutc()).strftime('%Y-%m-%d')

System Message: WARNING/2 (<string>, line 256)

Definition list ends without a blank line; unexpected unindent.

raise TypeError(repr(obj) + " is not JSON serializable")

es = rawes.Elastic("http://localhost:9200", json_encoder=encode_custom)

es.put('tweets/tweet/445', data={
'user' : 'dwnoble', 'post_date' : datetime(2012, 11, 12, 9, 45, 45, tzinfo=eastern_timezone), 'message' : 'Tweeting about elasticsearch'

System Message: WARNING/2 (<string>, line 264)

Definition list ends without a blank line; unexpected unindent.

})

es.get('tweets/tweet/445')['_source']['post_date'] # Returns: u'2012-11-12' ```

System Message: WARNING/2 (<string>, line 266); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 266); backlink

Inline interpreted text or phrase reference start-string without end-string.
JSON Decoding

Like with JSON encoding, a custom JSON decoder may be specified as well to parse elasticsearch results. A common use case here may be parsing ISO8601 dates to python datetime objects.

Index a document with a ISO8601 formatted date string:

```python import rawes es = rawes.Elastic()

System Message: WARNING/2 (<string>, line 278); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 278); backlink

Inline interpreted text or phrase reference start-string without end-string.
es.put('blogs/post/3', data={
'user' : 'dan', 'post_date' : '2013-7-04T23:14:53Z', 'title' : 'Elasticsearch 2', 'body' : 'More blogging about elasticsearch'

System Message: WARNING/2 (<string>, line 287)

Definition list ends without a blank line; unexpected unindent.
})

Define a custom JSON decoder: ```python import json import pytz import dateutil.parser

System Message: WARNING/2 (<string>, line 290); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 290); backlink

Inline interpreted text or phrase reference start-string without end-string.
class Iso8601JsonDecoder(json.JSONDecoder):

""" Automatically decode ISO8601 strings with key "post_date" to python datetime objects in UTC timezone """ def __init__(self):

System Message: ERROR/3 (<string>, line 301)

Unexpected indentation.
json.JSONDecoder.__init__(self, object_hook=self.dict_to_object)
def dict_to_object(self, d):
for k,v in d.iteritems():
if k == "post_date":
d[k] = dateutil.parser.parse(v)

System Message: WARNING/2 (<string>, line 307)

Definition list ends without a blank line; unexpected unindent.

return d

iso8601_json_decoder = Iso8601JsonDecoder()

```

System Message: WARNING/2 (<string>, line 312); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 312); backlink

Inline interpreted text or phrase reference start-string without end-string.

Now retrieve this document using our JSON decoder `python es.get("blogs/post/3")["_source"]["post_date"] # returns: # u'2013-7-04T23:14:53Z' es.get("blogs/post/3",json_decoder=iso8601_json_decoder.decode)["_source"]["post_date"] # returns: # datetime.datetime(2013, 7, 4, 23, 14, 53, tzinfo=tzutc()) es_default_decoder = rawes.Elastic(json_decoder=iso8601_json_decoder.decode) es_default_decoder.get("blogs/post/3")["_source"]["post_date"] # returns: # datetime.datetime(2013, 7, 4, 23, 14, 53, tzinfo=tzutc()) `

Error Handling

As of version 0.5, the rawes.Elastic constructor throws a rawes.elastic_exception.ElasticException any time elasticsearch returns an http status code of 400 or greater.

```python from rawes.elastic_exception import ElasticException es = rawes.Elastic('localhost:9200') try:

System Message: WARNING/2 (<string>, line 332); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 332); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: ERROR/3 (<string>, line 336)

Unexpected indentation.
es.get('invalid_index/invalid_type/123')

System Message: WARNING/2 (<string>, line 337)

Block quote ends without a blank line; unexpected unindent.
except ElasticException as e:
# since our index is invalid, this exception handler will run print e.result # prints: {u'status': 404, u'error': u'IndexMissingException[[invalid_index] missing]'} print e.status_code # prints: 404

System Message: WARNING/2 (<string>, line 343)

Definition list ends without a blank line; unexpected unindent.

```

System Message: WARNING/2 (<string>, line 343); backlink

Inline literal start-string without end-string.

System Message: WARNING/2 (<string>, line 343); backlink

Inline interpreted text or phrase reference start-string without end-string.
Thrift support

Thrift is supported for Python 2.x versions only. Before thrift will work with rawes, you must install the thrift python module `bash $ pip install thrift `

By default, connections on ports between 9500 and 9600 will use thrift `python import rawes es_thrift = rawes.Elastic('localhost:9500') `

If you are using thrift on a non standard port, specify a 'thrift://' url `python import rawes es_thrift = rawes.Elastic('thrift://localhost:8500') `

Run Unit Tests

rawes' unit tests require the python thrift module to run: `bash $ pip install thrift `

Run tests: `bash $ python -m unittest tests `

Run tests for Python 3 (no thrift tests, no need to "pip install thrift") `bash $ python3 -m unittest tests.py3k `

License

Apache 2.0 License

Contact

[@dwnoble](https://twitter.com/dwnoble)

History

0.5.3 (2013-7-4)
0.5.2 (2013-5-17)

System Message: WARNING/2 (<string>, line 399)

Title underline too short.

0.5.2 (2013-5-17)
----------------
0.5.1 (2013-5-15)

System Message: WARNING/2 (<string>, line 403)

Title underline too short.

0.5.1 (2013-5-15)
----------------
  • Added Python 3 http & https support (no thrift support yet for python 3) (thanks [adisbladis](https://github.com/adisbladis))
  • Removed python-dateutil requirement in favor of pytz (due to Python 3 not supporting python-dateutil)
0.5.0 (2013-4-25)

System Message: WARNING/2 (<string>, line 408)

Title underline too short.

0.5.0 (2013-4-25)
----------------
  • rawes now throws a rawes.elastic_exception.ElasticException on http status codes of >=400 by default. Removed rawes.Elastic "except_on_error" argument.
0.4.0 (2013-4-21)

System Message: WARNING/2 (<string>, line 412)

Title underline too short.

0.4.0 (2013-4-21)
----------------
  • Simplified Elastic constructor by removing connection_type param and incorporating in to url (Thanks [nvie](https://github.com/nvie)!)

  • Added optional **kwargs to Elastic constructor. Useful for specifying things like basic authentication or specific headers

    System Message: WARNING/2 (<string>, line 414); backlink

    Inline strong start-string without end-string.

0.3.6 (2013-1-8)
  • Fixed requests >1.0 incompatibilities, changed requirement back to 'requests>=0.11.1'
0.3.5 (2013-1-8)
  • Timeout bug fix for HTTP and Thrift
0.3.4 (2013-1-7)
  • Added 'except_on_error' boolean option to rawes.Elastic constructor. If set to True, rawes.Elastic will throw an exception when elasticsearch returns a status code of >= 400 (i.e., when there is an error)
0.3.3 (2012-12-21)
  • Restricted "requests" requirement from >=0.11.1 to '>=0.11.1, <1.0.0'
0.3.2 (2012-12-11)

System Message: WARNING/2 (<string>, line 433)

Title underline too short.

0.3.2 (2012-12-11)
----------------
0.3.1 (2012-11-26)

System Message: WARNING/2 (<string>, line 438)

Title underline too short.

0.3.1 (2012-11-26)
----------------
  • Changed python-dateutil version dependency from ==2.1 to >=1.0
0.3 (2012-11-12)
0.2 (2012-08-10)
  • Current Release

Subscribe to package updates

Last updated Jul 8th, 2013

Download Stats

Last month:1

What does the lock icon mean?

Builds marked with a lock icon are only available via PyPM to users with a current ActivePython Business Edition subscription.

Need custom builds or support?

ActivePython Enterprise Edition guarantees priority access to technical support, indemnification, expert consulting and quality-assured language builds.

Plan on re-distributing ActivePython?

Get re-distribution rights and eliminate legal risks with ActivePython OEM Edition.