Data Types¶
DynamoDB supports three different data types: STRING, NUMBER, and BINARY. It also supports sets of these types: STRING_SET, NUMBER_SET, BINARY_SET.
You can use these values directly for the model declarations, though they require an import:
from flywheel import Model, Field, STRING, NUMBER
class Tweet(Model):
userid = Field(type=STRING, hash_key=True)
id = Field(type=STRING, range_key=True)
ts = Field(type=NUMBER, index='ts-index')
text = Field(type=STRING)
There are other settings for type that are represented by python
primitives. Some of them (like unicode
) are functionally equivalent to the
DynamoDB option (STRING
). Others, like int
, enforce an additional
application-level constraint on the data. Each option works transparently, so a
datetime
field would be set with datetime
objects and you could query
against it using other datetime
‘s.
Below is a table of python types, how they are stored in DynamoDB, and any
special notes. For more information, the code for data types is located in
types
.
PY2 Type | PY3 Type | Dynamo Type | Notes |
---|---|---|---|
unicode | str | STRING | Basic STRING type. This is the default for fields |
str | bytes | BINARY | Binary data, (serialized objects, compressed data, etc) |
int/long | int | NUMBER | Enforces integer constraint on data |
float | NUMBER | ||
Decimal | NUMBER | ||
set | *_SET | This will use the appropriate type of DynamoDB set | |
bool | BOOL | ||
datetime | NUMBER | Stored with UTC timezone. See
DateTimeType for more. |
|
date | NUMBER | ||
dict | MAP | ||
list | LIST |
If you attempt to set a field with a type that doesn’t match, it will raise a
TypeError
. If a field was created with coerce=True
it will first
attempt to convert the value to the correct type. This means you could set an
int
field with the value "123"
and it would perform the conversion for
you.
Note
Certain fields will auto-coerce specific data types. For example, a
bytes
field will auto-encode a unicode
to utf-8 even if
coerce=False
. Similarly, a unicode
field will auto-decode a
bytes
value to a unicode string.
Warning
If an int
field is set to coerce values, it will still refuse to drop
floating point data. This has the following effect:
>>> class Game(Model):
... title = Field(hash_key=True)
... points = Field(type=int, coerce=True)
>>> mygame = Game()
>>> mygame.points = 1.8
ValueError: Field 'points' refusing to convert 1.8 to int! Results in data loss!
Set types¶
If you define a set
field with no additional parameters
Field(type=set)
, flywheel will ensure that the field is a set, but
will perform no type checking on the items within the set. This should work
fine for basic uses when you are storing a number or string, but sets are able
to contain any data type listed in the table above (and any custom type you declare). All you have to do is specify it in the
type
like so:
from flywheel import Model, Field, set_
from datetime import date
class Location(Model):
name = Field(hash_key=True)
events = Field(type=set_(date))
If you don’t want to import set_
, you can use an equivalent expression with
the python frozenset
builtin:
events = Field(type=frozenset([date]))
Field Validation¶
You can apply one or more validators to a field. These are functions that enforce some constraint on the field value beyond the type. Unlike the type checking done above, the validation checks are only run when saving to the database. An example:
class Widget(Model):
id = Field(type=int, check=lambda x: x > 0)
To apply multiple validation checks, pass them in as a list or tuple:
def is_odd(x):
return x % 2 == 1
def is_natural(x):
return x >= 0
class Widget(Model):
odd_natural_num = Field(type=int, check=(is_odd, is_natural))
There is a special case for enforcing that a field is non-null, since it is a common case:
username = Field(nullable=False)
The nullable=False
will generate an additional check to make sure the value
is non-null.
Custom Types¶
You can define your own custom data types and make them available across all of
your models. All you need to do is create a subclass of
TypeDefinition
. Let’s make a type that will
store any python object in pickled format.
from flywheel.fields.types import TypeDefinition, BINARY, Binary
import cPickle as pickle
class PickleType(TypeDefinition):
type = pickle # name you use to reference this type
aliases = ['pickle'] # alternate names that reference this type
ddb_data_type = BINARY # data type of the field in dynamo
def coerce(self, value, force):
# Perform no type checking because we can pickle ANYTHING
return value
def ddb_dump(self, value):
# Pickle and convert to a Binary object
return Binary(pickle.dumps(value))
def ddb_load(self, value):
# Convert from a Binary object and unpickle
return pickle.loads(value.value)
Now that you have your type definition, you can either use it directly in your code:
class MyModel(Model):
myobj = Field(type=PickleType())
Or you can register it globally and reference it by its type
or any
aliases
that were defined.
from flywheel.fields.types import register_type
register_type(PickleType)
class MyModel(Model):
myobj = Field(type='pickle')