Moving from AppEngine and taking your data with you

This article appeared first at techtavern.wordpress.com.

AppEngine and its companion Datastore have been of great value for our application. Nevertheless, time has come to move to another cloud, which also means to face the annoying fact that all our data is stored within a closed, isolated, black box within Googles infrastructure and accessible only through Googles proprietary AppEngine API.

This article explains how we managed to download all data from Datastore and prepared it for another cloud.

Our requirements

We considered 4 major requirements:

  1. Preserve the data as is (which means losing no information due to data representation issues).
  2. Preserve document oriented and non-relational nature of data (which means that data will not easily fit into tables).
  3. Preserve parent-child relationship between entities (which means not moving related entities away).
  4. Support very large amount of data (which means it won’t fit into CSV files).
  5. Allow to analyze data before importing it to the new cloud (which means to enable doing some basic data engineering tasks).

Possible approaches

There are 3 major approaches discussed on Internet to extract data from Datastore:

  1. Import data to Google BigQuery.
  2. Offer an “export” feature within the application.
  3. Export a backup to Google Storage.

Our investigation on each approach

Moving data to BigQuery sounds cheerful, but we would still be stuck within Google infrastructure. And it will not guarantee to preserve the original data due to forced format conversions and limitations to represent entity relationship.

We do not consider the implementation of an export feature as straightforward task. It would be time consuming do provide a correct implementation able to handle all scenarios of entities. These implementation would be difficult to test, to monitor and to validate, as it will run only inside AppEngine.

There are several opensource implementations of libraries that provide generic features to export data from within the application. Many of them have not been maintained anymore for several years. They would require certain amount of understanding and reverse engineering to assure they fit our purpose.

Google Cloud console states that the backup by Datastore console is encoded as standard LevelDB. But no LevelDB implementation was able to read it. But people reported that these files are encoded as Google Protocol Buffers and that there have been legacy libraries able to read these files.

There is only rare information about what is contained within the output files. And there are not official libraries that claim to understand them. Fortunately, there is previous work reported at Stackoverflow. This project at Github from Venryx demonstrates how to use Python legacy libraries from Appengine to read the files, converting Protocol Buffer streams into entities represented as Python objects (mostly dictionaries with additional Datastore metadata).

Our strategy

We decided to explore the former approach: to export a backup to Google Storage and convert this backup to a understandable format.

It has the convenience of breaking the problem into three smaller, independent ones:

  1. Export a backup from Datastore to Google Storage.
  2. Transfer the backup from Google Storage to a storage outside Google infrastructure.
  3. Convert the proprietary representation into an understandable representation;
  4. Import the data into the new cloud.

Step a) and b) need to be executed only once and are straightforward. And we incur the expensive Datastore and Storage billing only once. As result, we get a ’local’ copy we can investigate and reverse engineer step by step, without recurring to Google infrastructure again.

By converting the data to the understandable representation, we were able to analyze the data and to decide the best strategy to import it into the destination cloud. We were also able to validate the converted data before assuming the burden of importing it into the new cloud.

For the understandable representation, we decided to use JSON. While this representation is not as compact and as efficient. But it is a document oriented solutions. Tt allows to represent the hierarchical nature of entities. And it allows inspection by well-known open source tools. Therefore, we consider it a perfect fit for our requirements.

Our further investigation

Datastore does export the data into many (maybe tens of thousands of) files called “output-i”, which seem to be about 1-10MB. Probably, the size of each file will depend on the number of entities stored within the file and the size of each entity. We suppose that each file contains the result of a short time map-reduce background job, which seems to be the solution used by Datastore export feature.

Our investigation also revealed that, apparently, entities within the same output file are potentially parent-child related. At least from our data, we could not observe related entities spread into multiple output files. And root entities within the output file are potentially of the same kind. However, child entities are potentially of different kind, depending on how data was modeled.

Based on these considerations, we felt safe to decode each file separately. This will allow us to run the decoding in map-reduce style: convert all output files in parallel and merge the results from the same root entities, which gracefully satisfied the challenge to support very large amount of data.

Our first approach

We started with this project at Github from Venryx, which proposes to convert the backup files into a single json file. By following the minimal instruction, we were able to convert the backup into JSON, but the implementation has many drawbacks:

  1. It produces a single and enormous JSON file.
  2. It loads all entities into memory before writing them to the file.

Therefore, this approach is not suitable for very large amount of data. There might not be enough resources to load everything into memory. If something fails, the entire conversion fails without any intermediary result to resume after fixing the failure. It does not allow processing the files in parallel processes (targeting a map reduce approach). It is nearly impossible to load, analyze and validate the enormous JSON file.

The JSON contains a dictionary where each key is the entity kind. And the value is an array of entities of the respective kind.

Further, we noticed that:

  1. Encoding of string fields are UTF-8, but with apparent escape codes written as ascii.
  2. Unable to decode fields that contain embedded entities, references (keys) to other entities, binary data and many other types.
  3. All entities are handled as root entities.

Our improved approach

We changed Venryx’s implementation to handle one backup file at time instead of all files. And to produce a JSON file for each backup file. This allows running the conversion in parallel and to restart the conversions that failed. The JSON files are now small.

We also changed the implementation to write JSON preserve the parent child relation from the Datastore. The JSON now contains a dictionary where each key is the root entity kind. And the value is an array of entities of the respective kind. Within each entity, we add an additional dictionary to map child entities. Again, each key is the entity kind and the value is an array of entities of the respective kind. This approach repeats recursively.

We added additional handling to fields of no trivial type. The Protocol Buffer approach used by Datastore does not distinguish composed types or binary types. They are all handled ascii strings. One must know the data model to decide if a string field is really a string, a binary value or an encoded value (for example, embedded entity). Therefore, it will not be possible to create a generic approach. The code needs to inspect each entity and decide on its kind and fields, how to decode each of them.

To decode references to other entities, the API will return a field of type “Key”. Therefore, we iterate over all field and convert those of type Key. Remember that Datastore may use and ID or a Name to identify the entity.

def keyToJson(key):
  ds_kind = key.kind()
  ds_name = key.name()
  ds_id = key.id()
  if ds_name:
    return { 'kind': ds_kind, 'name': ds_name }
  if ds_id:
    return { 'kind': ds_kind, 'id': ds_id }

def decodeReference(entity_dict):
  for k in entity_dict:
    v = entity_dict[k]
    if v and type(v) is datastore_types.Key:
      entity_dict[k] = keyToJson(v)

To decode binary types. Remember that fields may be present, but may be None. In this example, the JSON will display the binary data as base64 encoded. Any other encoding may will be fine, except true binary encoding, as the respective ascii characters are not supported by the JSON writer. Note that your method needs to know the model and look for every pair of kind/field. This example decodes the salted/hashed password, that is stored as a binary field.

def decodeBinary(entity_kind, entity_dict):
  if entity_kind == 'User' and 'password' in entity_dict:
    v = entity_dict['password']
    if v:
      entity_dict['password'] = v.encode('base64').replace('\n', '')

One must implement many other decoders for each possible binary/composed type that is not supported by the Protocol Buffer model of Datastore. You will find all embedded/binary types at google.appengine.api.datastore_types (from google.appengine.api import datastore_types).

To decode embedded types, we approached similarly to binary types. We need to consider that the string contains binary data, that is the Protocol Buffer representation of the embedded entity. It took some hours to discover how to create an embedded entity from this string, but it is quite simple. We convert the entity to dict in order to remove all metadata from Datastore. This example decodes home address, that is stored as an embedded entity of the Client entity.

def decodeEmbedded(entity_kind, entity_dict):
  if entity_kind == 'Client' and 'address' in entity_dict:
    v = entity_dict['address']
    if v:
      entity = datastore.Entity.FromPb(v)
      entity_dict['address'] = dict(entity)

To fix the UTF-8 encoding issues, we changed the JSON dump to:

out = io.open(outFilePath, 'w', encoding='utf8')
out.write(json.dumps(value, default=JsonSerializeFunc, ensure_ascii=False, indent=2))

The resulting main loop:

# For a given entity key, returns the dictionary where to add fields of the entity.
# It creates all nodes on the tree according to the path described by the key.
# For example, a root entity will be mapped to jsonTree/kind/id_or_name
# A child entity will be mapped to jsonTree/root_kind/root_id_or_name/kind/id_or_name
# and so on.
def get_dest_dict(key, jsonTree):
  parent = key.parent()
  if parent is None:
    kind = key.kind()
    id_or_name = key.id_or_name()
    if kind not in jsonTree:
      jsonTree[kind] = {}
    if id_or_name not in jsonTree[kind]:
      jsonTree[kind][id_or_name] = {}
    return jsonTree[kind][id_or_name]
  else:
    jsonTree2 = get_dest_dict(key.parent(), jsonTree)
    kind = key.kind()
    id_or_name = key.id_or_name()
    if kind not in jsonTree2:
      jsonTree2[kind] = {}
    if id_or_name not in jsonTree2[kind]:
      jsonTree2[kind][id_or_name] = {}
    return jsonTree2[kind][id_or_name]

def Start():
  jsonTree = {}
  files = sorted(os.listdir(sourceDir))
  # or a list of files given by command line arguments

  for filename in files:
    if not filename.startswith("output-"): continue
    print("Reading from:" + filename)

    inPath = os.path.join(sourceDir, filename)
    raw = open(inPath, 'rb')
    reader = records.RecordsReader(raw)
    for recordIndex, record in enumerate(reader):
      # Get the protocol buffer representation of the next entity
      entity_proto = entity_pb.EntityProto(contents=record)
      # Convert protocol buffer representation to entity
      entity =  datastore.Entity.FromPb(entity_proto)
      # Convert entity to dict, to remove Datastore metadata
      dict2 = dict(entity)
      # Do not convert unwanted entity
      if kind == 'UnwantedEntity': continue
      # Some specific conversion needed for User entity (just for example).
      if kind == 'User':
         pass
      # Decode fields of binary/composed types
      decodeBinary(kind, dict2)
      decodeEmbedded(kind, dict2)
      decodeReference(dict2)

      key = entity.key()
      dest_dict = get_dest_dict(key, jsonTree)
      dest_dict.update(dict2)

    outFilePath = os.path.join(destDir, filename+'.json')
    out = io.open(outFilePath, 'w', encoding='utf8')
    out.write(json.dumps(value, default=JsonSerializeFunc, ensure_ascii=False, indent=2))
    out.close()
    print("JSON file written to: " + outFilePath)
    jsonTree = {}