ARGOS SERVER INSTALLATION NOTES alpha 0.7 release; Don Gilbert; March 2004 This ARGOS system is designed for automatic installation and updates of FlyBase and other genome information servers. This includes scripts, configurations, data, and Unix binaries for all needed programs except Perl, Java and rsync. Rsync is used as the primary distribution program. ARGOS currently forms the genome database/web server infrastructure for the public genome databases at Indiana University including FlyBase.net, euGenes.org, and the nascent Daphnia water flea database. Basic development of this system continues, as part of the GMOD group project for generic organism genome databases, but it is now usable for replication of genome databases, with flexibility to add new such databases. Uses of automated database server distribution include local cluster load distribution, world-wide mirror sites for rapid and redundant access, developing bio-grid uses, institution/company servers for local projects and data mining. An automatable mirroring system differs from the method of providing software and data downloads by FTP in that packages of data and software in this system are kept up-to-date without human intervention. Similar package management systems such as RPM, PacMan, the Grid replication tools, and others share common functions, but have proved not quite suitable yet for bio-database distribution. Server packages are currently hosted at rsync://flybase.net/argos/ Web notes are at http://eugenes.org/argos/ or http://flybase.net/argos/ Installation of a full FlyBase or other server will require several gigabytes of disk space. Some of these hosts have this server running http://flybase.bio.indiana.edu:8081/ -- solaris http://iubio.bio.indiana.edu:8081/ -- solaris http://chervil.bio.indiana.edu:8081/ -- linux http://marmot.bio.indiana.edu:8081/ -- macosx See also install/argos-system-management.txt and background reading in the docs/ folder. REQUIREMENTS ============ Unix systems supported for binary packages: Sun Solaris v8 & v9 Intel Linux (redhat v8,9 tested) Apple MacOSX v10.2 These packages need to be and commonly are preinstalled except for Solaris v8 Perl v5.6 or later - http://www.perl.com/ Java v1.3 or later - http://java.sun.com/ rsync v2.5 or later - http://rsync.samba.org/ PREPARATION =========== This system can be run from any Unix account with disk space and ability to use Unix ports. For a public server that is automatically updated, we recommend 1. Create a normal unix account just for this service (e.g. argosadm) locating it on a disk with room for data (see in install.cgi a list of disk use). The ARGOS_ROOT path will be set to this account folder or one inside it. Replication will manage files inside ARGOS_ROOT. 2. Optionally create a second, underpriviledged account (like nobody) for the web server processes. This will require you run the apache server as root. It is not needed for lightly used servers, but recommended for heavy use. INSTALLATION ============ Start with install.cgi web form ------------------------------------ http://flybase.net/argos/install/install.cgi This is the recommended starting point. The web form handles Steps 0 and 2 below for producing the local configuration for your system, and lists other necessary steps (as below). Summary of steps ---------------- 1. Download install script : http://flybase.net/argos/install/installargos.pl 2. Bootstrap installation : From command line, run this, changing -root= to your preferred location 'perl installargos.pl -root=/bio/argos/ -install -v' This creates install/ and a few other folders at the root path and fetches default configuration files. 3. Configure : Use the install.cgi web form to create install.conf.local with paths and port numbers for your system. This configuration file for your system needs to be placed in the install/ folder created in (2). 4. Install a full data and software set : 'cd /bio/argos; install/install -install -v' OR 'install/install -install -v >& log.install &' to run in background. This can take a while. Check installation files and log.install for errors when this finishes. 5. Start the web server: 'install/run-apache start' The Argos web server should then be available at http://localhost:8081/ with installed packages at the ports that you configured it to use. Read the logs/ folder for the *.err error logs to explain if this isn't working. 6. Read the argos-system-management.txt document. As appropriate, add cron tasks to this argos account for updating and server restarting. Use command-line installargos.pl script ------------------------------------------- This Perl script handles the bulk of data installation and updates. installargos.pl [ options ] -info|install|list|update|reconfigure The -info and -list commands make no changes to disk. All commands are benign if used with -n[orun] option. Some other installargos options: installargos.pl -info == show packages information installargos.pl -list -pack flybase == list all files in package, and required packages, from distribution servers. installargos.pl -update == update installed packages on frequent basis installargos.pl -no -install == view steps without doing them installargos.pl -root=/usr/local/argos == use new root path HISTORY ============ winter 2004: Argos now is public server for FlyBase-NG, euGenes fall 2003 : updates for FlyBase public server including slave servers for BLAST load balancing service; GBrowse (fb) and others summer 2003: added genome services: euGenes.org, daphnia-wFleaBase, centaurbase (test) may 2003 : renamed from flybaseng to argos march 2003 : changed preliminary biodb to flybase-ng WORKED EXAMPLE ============== Installing all of flybase-ng may be time consuming - it still needs an easier installation script. Don't install as root -- but as a regular user, such as yourself. The basic step is to copy the installargos.pl perl script to a local folder and run are below in my test. # --------- # STEP 1 -- DOWNLOAD # http://flybase.net/flybase-ng/install/ installargos.pl # to a local folder where you want to install. # --------- dghome2% perl installargos.pl Usage: installargos.pl [ options ] -info|install|list|update|reconfigure ** 1. Run installargos.pl -install [-root=/bio/argos] == bootstrap install ** 2. Edit /bio/argos/install/install.conf.local to set configuration. 3. Run /bio/argos/install/install -install == copy server packages Packages selected from packages.conf will by copied from servers. Start web servers with /bio/argos/install/run-apache Web notes are at http://flybase.net/flybase-ng/ -- details omitted -- configuration= PACKAGES=argos-install, ROOT=/bio/argos packages= argos-install # --------- # STEP 2 -- run with parameters # IF YOU WANT TO USE default path /bio/biodb, skip the -root=... option # USE THE '-n' option to see what it will do without doing it. # --------- dghome2% perl installargos.pl -root=/Users/gilbertd/temp/argos -install -v config file missing: ./packages.conf config file missing: ./install.conf config file missing: ./install.conf.local config file missing: ./reconfigure.conf # INSTALL server directories mkdir -p /Users/gilbertd/temp/argos/install # OS=apple-powerpc-darwin # INSTALL package=argos-install # INSTALL rsync://flybase.net/argos/install/ rsync -a --exclude-from=/Users/gilbertd/temp/argos/install/rsync.exclude.local rsync://flybase.net/argos/install/ /Users/gilbertd/temp/argos/install/ cp -p /Users/gilbertd/temp/argos/install/install.conf /Users/gilbertd/temp/argos/install/install.conf.local perl -pi -e s,^ROOT=\S+,ROOT=/Users/gilbertd/temp/argos,; /Users/gilbertd/temp/argos/install/install.conf.local mkdir -p /Users/gilbertd/temp/argos/common/ ln -s ./systems/apple-powerpc-darwin /Users/gilbertd/temp/argos/common/system-local # RECONFIGURE packages dghome2% ls common/ install/ installargos.pl* # --------- # STEP 3 # EDIT BY HAND install/install.conf.local # OR see for web form configuring (in progress) # http://flybase.net/flybase-ng/install/ install.cgi # --------- # CHANGE: # path to installation ROOT=/bio/biodb # packages to install PACKAGES="biodb-root flybase" FLYBASE_CONF=#FLYBASE_CONF # TO THESE (or other desired): # path to installation ROOT=/Users/gilbertd/argos1 # packages to install PACKAGES="biodb-root flybase" FLYBASE_CONF= # --------- # STEP 4 -- run after bootstrap as install/install # -- check configuration values match above # --------- dghome2% install/install Usage: install/installargos.pl [ options ] -info|install|list|update|reconfigure 1. Run install/installargos.pl -install == bootstrap install 2. Edit /bio/biodb/install/install.conf.local to set configuration. 3. Run /bio/biodb/install/install -install == copy server packages ... configuration= APACHE_PORT=8081, DAPHNIA_CONF=#DAPHNIA_CONF, EUGENES_CONF=#EUGENES_CONF, EUGENES_PORT=7072, FLYBASE_CONF=, FLYBASE_PORT=7082, GMODWEB_CONF=#GMODWEB_CONF, PACKAGES=biodb-root flybase, PGPORT=7302, ROOT=/Users/gilbertd/argos1, TOMCAT_FLYBASE=7882, TOMCAT_PORT=7880 --------- # ACTUALLY DO THE installation for flybase... dghome2% install/install -install -v # if you want list of all files copied, use -rsopt=-v # SINCE it takes a while, dghome2% install/install -install -v >& log.install & # --------- # STEP 5 -- check installation, run web server: # --------- install/run-apache install/run-apache start # check result in logs/ folder ---------