Welcome, guest | Sign In | My Account | Store | Cart

Python implementation for generating Tiny URL- and bit.ly-like URLs.

Python, 126 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
'''
Short URL Generator
===================

Python implementation for generating Tiny URL- and bit.ly-like URLs.

A bit-shuffling approach is used to avoid generating consecutive, predictable 
URLs.  However, the algorithm is deterministic and will guarantee that no 
collisions will occur.

The URL alphabet is fully customizable and may contain any number of 
characters.  By default, digits and lower-case letters are used, with 
some removed to avoid confusion between characters like o, O and 0.  The 
default alphabet is shuffled and has a prime number of characters to further 
improve the results of the algorithm.

The block size specifies how many bits will be shuffled.  The lower BLOCK_SIZE 
bits are reversed.  Any bits higher than BLOCK_SIZE will remain as is.
BLOCK_SIZE of 0 will leave all bits unaffected and the algorithm will simply 
be converting your integer to a different base.

The intended use is that incrementing, consecutive integers will be used as 
keys to generate the short URLs.  For example, when creating a new URL, the 
unique integer ID assigned by a database could be used to generate the URL 
by using this module.  Or a simple counter may be used.  As long as the same 
integer is not used twice, the same short URL will not be generated twice.

The module supports both encoding and decoding of URLs. The min_length 
parameter allows you to pad the URL if you want it to be a specific length.

Sample Usage:

>>> import short_url
>>> url = short_url.encode_url(12)
>>> print url
LhKA
>>> key = short_url.decode_url(url)
>>> print key
12

Use the functions in the top-level of the module to use the default encoder. 
Otherwise, you may create your own UrlEncoder object and use its encode_url 
and decode_url methods.

Author: Michael Fogleman
License: MIT
Link: http://code.activestate.com/recipes/576918/
'''

DEFAULT_ALPHABET = 'mn6j2c4rv8bpygw95z7hsdaetxuk3fq'
DEFAULT_BLOCK_SIZE = 24
MIN_LENGTH = 5

class UrlEncoder(object):
    def __init__(self, alphabet=DEFAULT_ALPHABET, block_size=DEFAULT_BLOCK_SIZE):
        self.alphabet = alphabet
        self.block_size = block_size
        self.mask = (1 << block_size) - 1
        self.mapping = range(block_size)
        self.mapping.reverse()
    def encode_url(self, n, min_length=MIN_LENGTH):
        return self.enbase(self.encode(n), min_length)
    def decode_url(self, n):
        return self.decode(self.debase(n))
    def encode(self, n):
        return (n & ~self.mask) | self._encode(n & self.mask)
    def _encode(self, n):
        result = 0
        for i, b in enumerate(self.mapping):
            if n & (1 << i):
                result |= (1 << b)
        return result
    def decode(self, n):
        return (n & ~self.mask) | self._decode(n & self.mask)
    def _decode(self, n):
        result = 0
        for i, b in enumerate(self.mapping):
            if n & (1 << b):
                result |= (1 << i)
        return result
    def enbase(self, x, min_length=MIN_LENGTH):
        result = self._enbase(x)
        padding = self.alphabet[0] * (min_length - len(result))
        return '%s%s' % (padding, result)
    def _enbase(self, x):
        n = len(self.alphabet)
        if x < n:
            return self.alphabet[x]
        return self._enbase(x / n) + self.alphabet[x % n]
    def debase(self, x):
        n = len(self.alphabet)
        result = 0
        for i, c in enumerate(reversed(x)):
            result += self.alphabet.index(c) * (n ** i)
        return result

DEFAULT_ENCODER = UrlEncoder()

def encode(n):
    return DEFAULT_ENCODER.encode(n)

def decode(n):
    return DEFAULT_ENCODER.decode(n)

def enbase(n, min_length=MIN_LENGTH):
    return DEFAULT_ENCODER.enbase(n, min_length)

def debase(n):
    return DEFAULT_ENCODER.debase(n)

def encode_url(n, min_length=MIN_LENGTH):
    return DEFAULT_ENCODER.encode_url(n, min_length)

def decode_url(n):
    return DEFAULT_ENCODER.decode_url(n)

if __name__ == '__main__':
    for a in range(0, 200000, 37):
        b = encode(a)
        c = enbase(b)
        d = debase(c)
        e = decode(d)
        assert a == e
        assert b == d
        c = (' ' * (7 - len(c))) + c
        print '%6d %12d %s %12d %6d' % (a, b, c, d, e)

Short URL Generator

Python implementation for generating Tiny URL- and bit.ly-like URLs.

A bit-shuffling approach is used to avoid generating consecutive, predictable URLs. However, the algorithm is deterministic and will guarantee that no collisions will occur.

The URL alphabet is fully customizable and may contain any number of characters. By default, digits and lower-case letters are used, with some removed to avoid confusion between characters like o, O and 0. The default alphabet is shuffled and has a prime number of characters to further improve the results of the algorithm.

The block size specifies how many bits will be shuffled. The lower BLOCK_SIZE bits are reversed. Any bits higher than BLOCK_SIZE will remain as is. BLOCK_SIZE of 0 will leave all bits unaffected and the algorithm will simply be converting your integer to a different base.

The intended use is that incrementing, consecutive integers will be used as keys to generate the short URLs. For example, when creating a new URL, the unique integer ID assigned by a database could be used to generate the URL by using this module. Or a simple counter may be used. As long as the same integer is not used twice, the same short URL will not be generated twice.

The module supports both encoding and decoding of URLs. The min_length parameter allows you to pad the URL if you want it to be a specific length.

Sample Usage:

>>> import short_url
>>> url = short_url.encode_url(12)
>>> print url
LhKA
>>> key = short_url.decode_url(url)
>>> print key
12

Use the functions in the top-level of the module to use the default encoder. Otherwise, you may create your own UrlEncoder object and use its encode_url and decode_url methods.

3 comments

J Singh 14 years, 4 months ago  # | flag

A minor error: Line 41 should be

return self.enbase(x/n) + self.alphabet[x%n]

Thanks for the recipe. It is well conceived, well written code.

Greg 14 years, 1 month ago  # | flag

is there a way to make it work with a url? i can shorten integers all day long, but when i actually enter a url i get this:

>>> import short_url
>>> link = '/home/gmilby/public_html/syrbotwebdesign.com/scripts/playground'
>>> url = short_url.encode_url(link)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "short_url.py", line 66, in encode_url
    return DEFAULT_ENCODER.encode_url(n, min_length)
  File "short_url.py", line 15, in encode_url
    return self.enbase(self.encode(n), min_length)
  File "short_url.py", line 19, in encode
    return (n & ~self.mask) | self._encode(n & self.mask)
TypeError: unsupported operand type(s) for &: 'str' and 'long'

i kept shaving off the beginning (http:, then // then...) same error everytime.

Luana Reis Azeredo 12 years, 6 months ago  # | flag

LIC (Computational Intelligence Laboratory), Francisco Reinaldo, Luana Azeredo:

test in version 3.2:

>>> import short_url
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    import short_url
  File "C:\Python32\short_url.py", line 78
    print '%6d %12d %s %12d %6d' % (a, b, c, d, e)
                               ^
SyntaxError: invalid syntax
>>>

changes:

line 11: self.mapping = list(range(block_size))

method: _enbase(self, x): x = int(x) #convert x

line 79: print ('%6d %12d %s %12d %6d' % (a, b, c, d, e))

Created by Michael Fogleman on Wed, 30 Sep 2009 (MIT)
Python recipes (4591)
Michael Fogleman's recipes (4)

Required Modules

  • (none specified)

Other Information and Tasks