transfer_vn Documentation
User Guide
Initial setup
Initialize the sample YAML file in your HOME directory and edit with your local details. The YAML file is self documented:
transfer_vn --init .evn_your_site.yaml
editor $HOME/.evn_your_site.yaml
Create the database and tables:
transfer_vn --db_create --json_tables_create --col_tables_create .evn_your_site.yaml
Running the application
After editing YAML configuration file, you should proceed in 3 stages:
Transfer of all historical data
transfer_vn --full
. Note: the probability of a blocking error during this large transfer is not negligible. In this case, it is possible to resume the complete download by limiting the taxonomic groups and/or the start and end dates.Schedule update tasks, based on recurrences defined in the YAML configuration file:
transfer_vn --schedule
Regular incremental update, for example in an hourly crontab:
transfer_vn --update
Transfers overwrite data already downloaded. It is therefore not a problem to
restart transfer_vn --full
on part of the database.
Beware that, depending on the volume of observations, this can take several hours. We recommend starting with a small taxonomic group first:
transfer_vn --full .evn_your_site.yaml
After this full download, data can be updated. For observations, only new,
modified or deleted observations are downloaded. For other controlers, a full
download is always performed. Each controler runs on its own schedule,
defined in the YAML configuration file. This step needs to be performed
after each --full
execution or YAML file modification. To create or update,
after modifying the configuration file, the schedule:
transfer_vn --schedule .evn_your_site.yaml
Once this is done, you can update the database with new observations:
transfer_vn --update .evn_your_site.yaml
This can be done by cron, every hour for example. At each run, all scheduled
tasks are performed. Note: you must wait until the first scheduled task has
expired for a transfer to be carried out. With the default schedule, you must
therefore wait for the next round hour --schedule
. It must run at least
once a week. The virtual environment must be activated in the cron job, for
example:
0 * * * * echo 'source client_api_vn/env_VN/bin/activate;cd client_api_vn/;transfer_vn --update .evn_your_site.yaml --verbose'| /bin/bash > /dev/null
Command-line options - transfer_vn
The application runs as:
transfer_vn options config
where:
options command line options described below
config YAML file, located in $HOME directory, described in sample file
- -h, --help
Prints help and exits
- --version
Print version number
- --verbose
Increase output verbosity
- --quiet
Reduce output verbosity
- --init
Initialize the YAML configuration file
- --db_drop
Delete if exists database and roles
- --db_create
Create database and roles
- --json_tables_create
Create or recreate JSON tables in import schema
- --col_tables_create
Create or recreate colums based tables
- --migrate
Migrates the JSON import schema to latest version
- --full
Perform a full download
- --update
Perform an incremental download
- --schedule
Create or update the incremental update schedule
- --status
Print downloading status (schedule, errors…)
- --count
Count observations by site and taxo_group
- --profile
Gather and print profiling times