Main Page | Recent changes | Edit this page | Page history

Printable version | Disclaimers | Privacy policy

 Not logged in
Log in | Help
 

YaM

From DGC Wiki

Contents

YaM is a software development framework with tools for facilitating the rapid development of software in a concurrent software development environment [1]. Team Caltech uses YaM to implement its configuration management process. This page collects the essential information that you need to know in order to make use of YaM.

More info:

[edit]

Getting started

This section contains a quick guide to using YaM. It makes use of the play project, which has been set up for testing. Within this project there is a copy of the "follow" package, which allows you to run a controller on Alice (either in hardware or in simulation). If any of the commands listed here don't work, please send e-mail to Richard Murray or fix the wiki page once you figure out what is wrong.

In this tutorial, we will create a new module that prints your name on the screen and then integrate this new module into the "follow" package. We will then generate a new release of the package. The process that we use for this is consistent with the Team Caltech configuration management process, with the role of the developer and the build manager both being done by a single person.

[edit]

Setting up your environment

YaM makes use of several environment variables to keep track of the project you are working on. You should set up the following environment variables in your shell initialization script (.cshrc, .bash.rc, etc):

[edit]

Creating a bug/task request

The first step in the configuration management process is to create a request for a change. This is not strictly part of YaM, but is an important part of the way we keep track of software development. To create a change request, you should do the following:

  1. Log in to the bugzilla system
  2. Create a new bug with a summary that reads "Add user <your name> to follow display". Enter a short indicating that you are working through the YaM tutorial
  3. Assign the bug to yourself and accept it, so that we know you are working on it
  4. Set the "Status" flag to "In Progress" (not yet implemented)
[edit]

Checking out a "sandbox"

stefano : Move this line at the beginning of the section. Everybody is realizing this too late!!

You might have to type your password a lot if you have not set up .ssh so that it knows which machines you can log in from automatically. To do this:

  1. run the command 'ssh-keygen -t dsa' on gclab and hit enter at the prompts.
  2. This will create a file '~/.ssh/id_dsa.pub' which you append to the file '~/.ssh/authorized_keys'- eg, run 'cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys'
  3. make sure that the permissions mode of the authorized_keys file is -rw-r--r-- (644)
  4. copy your .ssh folder to gcfield

You'll know this worked if you can ssh to any machine in the lab or alice from another machine in the lab or alice without typing your password. See ssh-keygen(1) for more info.

Source code development is done in a YaM sandbox. This is basically a copy of the code that you can use for adding new functionality to a package. We start by checking out a copy of the "follow-test" package, which consists of the modules that make up the follow package. To check out a sandbox, you should do the following:

  1. Check out the follow-test package using YaM 
    yam setup follow-test
  2. An editor window will pop up with the file config.yam opened up. This file should list the modules that are part of the package (including skynet, sparrow, adrive, etc) as well as the revisions (branches) of each module. You can exit the editor without making any changes.

This will create a subdirectory of the current directory with the name "follow-test" which has the code for the "follow-test package". If you cd into the 'src' subdirectory, you will see the code that is included in the package, which should consist of each of the modules listed above.

Once you have checked out the sandbox, it is good idea to document this in bugzilla so that we can keep track of where changes were implemented:

  1. Log in to the bugzilla system and find the bug you created
  2. Enter a comment saying that you have created "Rx-yy-user" to work on this feature.
[edit]

Making your changes

The current version of "follow-test" does not have the Makefiles set up yet, so you will need to manually make and install the modules. Do this by going into each subdirectory in the 'src' directory and typing "make install". You may need to install some modules before others to get the dependencies right (YaM makefiles will do this correctly once they are working). The following should work: 

# make -C src/sparrow install
# make -C src/skynet install
# make -C src/adrive install
# make -C src/asim

If you cd to the 'src/asim' directly, you should be able to run asim (simulator) and see a sparrow display come up. If you ge tan error about libserial, you may need to set up the load library path: 

export LD_LIBRARY_PATH=usr/local/lib      # for bash
setenv LD_LIBRARY_PATH /usr/local/lib      # for tcsh

At this point, you are ready to make modifications to the code. Pick some module and make a change that is visible (change a startup message or perhaps modify the sparrow display on asim or follow).

Once you have completed your changes, commit them to the respository using 'svn commit' as usual. When you are ready to create a new revision of your code (meaning that you want to release it to the build manager), use the command 

yam save <module>

where 'module' is one of 'asim', 'skynet', etc. This will create a new release of the module, incrementing the module number. It will also remove the source code from your sandbox, unless you recheck it out (by moving the module from a 'link' module back to a 'work' module. You should update your bugzilla bug, indicating that you have made the appropriate change and are done with your testing.

[edit]

Releasing a new package

stefano : This is not clear

For this section, you are operating as the build manager for the package. You should pretend that someone else committed the module changes and that you are now going to check those changes to make sure they are OK.

Start by checking out a fresh copy of the package from the repository. Make sure that you either explicitly specify the release numbers for the modules that you want to use. Once you have the package checked out, compile and test the code to make sure that it works. When you are done, use the command 

yam pkgnewrelease

to create a new release for the package.

[edit]

Using the DGC project

Once you have successfully committed a module and package release under the 'play' project, you are ready to switch to the 'dgc' project. You do this by changing the YAM_* environment variables. Under bash, put the following lines in your .bashrc file: 

export YAM_PROJECT=dgc
export YAM_PROJECT_CONFIG_DIR=/dgc/yam/dgc
export YAM_SITE=lab       # gclab 
#  export YAM_SITE=alice      # gcfield

You now need to decide what package you want to check out. Here are some possibilities:

At the current time the makefiles for building an entire package haven't been setup, so you will need to individually build the modules that are part of a given package. You do this by typing the following command from within the top level sandbox directory: 

make -C src/<module> -f Makefile.yam all
[edit]

Customizing YaM

Here are some links to notes for customizing YaM

[edit]

Where to put data files

We need to figure out where to put data files within YaM.

Principles

Proposed solution

File names

[edit]

Updating link modules

There is often a need to bump up a sandbox's link module configuration to latest releases of the link modules. This is especially needed when trying to release a module from a sandbox. While some of you may already be aware of this, the "yam config" command is a handy way to accomplish this. From the top-level of your sandbox you can run 

    yam config -update-links -o newconfig

This command will read in your existing YAM.config, will update all the link modules to their latest versions and will create the 'newconfig' file with the updated contents. Typically you can replace the existing YAM.config with 'newconfig' and relink the sandbox. That is, run 

         mv newconfig YAM.config
         yam relink

These steps will bump up your sandbox so all the link modules are pointing to the latest link modules. After testing your work modules, you should be able to release them.

yam-config has other options for transforming YAM.config files (or for creating them from scratch). You can learn about them using the -h option. Since this command does not make any changes to a sandbox, its harmless to experiment with.

[edit]

Project Setup

This section describes how to set up a new project. This is intended as documentation for YaM administrators, since for developers the projects you need should already be set up. The directions here are pretty much directly from the YaM wiki.

[edit]

Set up the MySQL database

  1. Create a new MySQL database for YaM 
    gc$ mysqladmin -u root -p create yamplay
    The convention is to prefix the project name with 'yam' so that we can tell that is what the database is used for.
  2. Create the database structure needed by YaM 
    gc$  mysql -u root -p yamplay < /mnt/dgc-lab/software/yam/YaM-xxx/sql/create-yyy.sql

    (replace xxx and yyy with the latest version information)

  3. Give the YaM user access to the database 
     gc$ mysql -u root -p
 mysql> GRANT SELECT,INSERT,UPDATE,ALTER ON yamplay.* TO 'yam@%' IDENTIFIED BY 'yamPassword';
 mysql> flush privileges;
 mysql> quit;

    You should replace yamPassword with the desired password for the database. The '%' is a wildcard that will allow access from any host that can talk to the mysql server (on gc, this is restricted to internal hosts by our filewall rules).

[edit]

Set up the YaM project

  1. Create the directories where your YaM configuration will exist and set the permissions to allow group setgid and write access. This must be done on gclab since that is where the files live. 
     gclab$ sudo mkdir /dgc/yam/play
 gclab$ sudo mkdir /dgc/yam/tmp
 gclab$ sudo chmod g+sw /dgc/yam/play
 gclab$ sudo chmod g+sw /dgc/yam/tmp
  2. Create a configuration file 'play.config' using the YaM gui. See instructions in config files for a list of what to create, or look at an existing configuration file 
    gc$ yam -admin gui -init
    (Note: the gui will open up an X11 window on your computer. Make sure that X11 forwarding is turned on (ssh -Y) if you are setting things up remotely.) Make sure that your configuration file is saved as 'play.config' and that the project directory is set to '/dgc/yam/play'. You should initially set the svn repository to be of the form 'file:///' since this is required for yam initialization (you can change it back later to be 'svn+ssh://host/'.
  3. Test the configuration to make sure that everything is set up properly 
    gc$ yam -testdb

    Note that you need to have the YAM_PROJECT and YAM_PROJECT_CONFIG_DIR environment variables set up properly in order for this command to complete successfully.

  4. Initialize the YaM project 
    yam -admin project -init

    Notes

    • On gc, yamSVN.pm was modified to use database type 'fsfs', which is apparently not the default. Need to use this to get the group permissions right.
    • After this command runs, you should check the permissions on the subversion database and make sure they are properly initialized for group access.
  5. You can now reset the subversion repository to be of type 'svn+ssh://host/' using the gui.

[edit]

Set up the initial release

  1. Create the new modules in your project 
    yam mkmodule moduleName
  2. Create a package that includes a set of modules and/or subpackages: 
    yam mkpackage -mods "module1 module2 subpkg1 subpkg2" packageName
  3. Create a sandbox and populate the modules with the files you have
[edit]

Setup notes

In order to get everything working when I set this up, I had to do a number of additional steps (with help from Abhi Jain):

  1. Change permissions of executables in src/SiteDefs/mkHome/shared and set svn:executable property 
    chmod +x *.sh *.pl killprocs yamroot

This appears to be fixed in R1-72i.

  1. Copy 'makefile-yam-tail.mk' to src/SiteDefs. This defines all of the default rules that YaM wants in Makefile.yam. This appears to be fixed in R1-72i.
  2. Set up src/SiteDenfs/mkHome/shared/yamNative.sh to recognize the OS that I was using.
    • Modified this on gc to detect RHEL4 and gentoo
[edit]

Installation notes

[edit]

Release 72a installation

[edit]

Installation under Mac OSX

Warning: These notes are written assuming you know something about how to configure YaM on a linux client. Probably too terse at this point to be generally useful. If you try these out, make sure you get things working on the play account before you do something to the DGC repository.

Install Fink:

Install YaM:

Install the following perl modules:

Verify YaM is working:

Configure YaM

That should be it. At this point you can run yam just like on gclab.

Troubleshooting:

[edit]

Required Third Party Software

[edit]

Troubleshooting

[edit]

If you have a problem

Please enter a bug report. [2] -> Product: Implement -> Component: YaM

Make sure you attach the log file to this bug report.

Retrieved from "http://kincora.cds.caltech.edu../../../y/a/m/YaM_9908.html"

This page has been accessed 838 times. This page was last modified 22:05, 4 March 2007 by Richard Murray. Based on work by Andrew Howard, Jessica Gonzalez, Chris Schantz, Kenny Oslund, Nhattrieu Duong, stefano di cairano, Peter Trautman and Melvin Flores and DGC Wiki user(s) Kenneth.


[Main Page]
Main Page
Alice Docs
DGC07
DGC05
Categories
Recent changes
Help
Edit this page
Discuss this page
Page history
What links here
Related changes
Special pages
Bug reports