transfer_vn Documentation

User Guide

Initial setup

Initialize the sample YAML file in your HOME directory and edit with your local details. The YAML file is self documented:

transfer_vn --init .evn_your_site.yaml
editor $HOME/.evn_your_site.yaml

Create the database and tables:

transfer_vn --db_create --json_tables_create --col_tables_create .evn_your_site.yaml

Running the application

After editing YAML configuration file, you should proceed in 3 stages:

  1. Transfer of all historical data transfer_vn --full. Note: the probability of a blocking error during this large transfer is not negligible. In this case, it is possible to resume the complete download by limiting the taxonomic groups and/or the start and end dates.

  2. Schedule update tasks, based on recurrences defined in the YAML configuration file: transfer_vn --schedule

  3. Regular incremental update, for example in an hourly crontab: transfer_vn --update

Transfers overwrite data already downloaded. It is therefore not a problem to restart transfer_vn --full on part of the database.

Beware that, depending on the volume of observations, this can take several hours. We recommend starting with a small taxonomic group first:

transfer_vn --full .evn_your_site.yaml

After this full download, data can be updated. For observations, only new, modified or deleted observations are downloaded. For other controlers, a full download is always performed. Each controler runs on its own schedule, defined in the YAML configuration file. This step needs to be performed after each --full execution or YAML file modification. To create or update, after modifying the configuration file, the schedule:

transfer_vn --schedule .evn_your_site.yaml

Once this is done, you can update the database with new observations:

transfer_vn --update .evn_your_site.yaml

This can be done by cron, every hour for example. At each run, all scheduled tasks are performed. Note: you must wait until the first scheduled task has expired for a transfer to be carried out. With the default schedule, you must therefore wait for the next round hour --schedule. It must run at least once a week. The virtual environment must be activated in the cron job, for example:

0 * * * * echo 'source client_api_vn/env_VN/bin/activate;cd client_api_vn/;transfer_vn --update .evn_your_site.yaml --verbose'| /bin/bash > /dev/null

Command-line options - transfer_vn

The application runs as:

transfer_vn options config

where:

options  command line options described below
config   YAML file, located in $HOME directory, described in sample file
-h, --help

Prints help and exits

--version

Print version number

--verbose

Increase output verbosity

--quiet

Reduce output verbosity

--init

Initialize the YAML configuration file

--db_drop

Delete if exists database and roles

--db_create

Create database and roles

--json_tables_create

Create or recreate JSON tables in import schema

--col_tables_create

Create or recreate colums based tables

--migrate

Migrates the JSON import schema to latest version

--full

Perform a full download

--update

Perform an incremental download

--schedule

Create or update the incremental update schedule

--status

Print downloading status (schedule, errors…)

--count

Count observations by site and taxo_group

--profile

Gather and print profiling times