SProject Log 06
I have completed the part of the documentation
which I was planning to write.
- 15 Nov 2005
We have mostly neglected our log during the past two weeks, so I'm going to try to summarize what our team has been working
on lately, for reference purposes.
The Monday after the meeting metioned in the previous entry we have indeed proceeded with the integration of our
code in the trunk. At the same time, we've found out that Sjoerd was no longer following the Swe course, so Rinse
took over his responsabilities, which meant implementing the move, copy and revert methods (the only things that
were still not supported by our storage layer). Soon after Rinse started working on these methods, it turned out
that we were quite optimistic when thinking that writing copy, move and revert shouldn't be much of a problem. It
actually turned out to be quite a difficult task, both in itself and in that a lot of our code stopped working properly
with the configurations of our virtual working copy introduced by executing copy, move or revert. Rinse and I have
probably spent somewhere around 10 hours (cumulated) discussing various issues that kept coming up, and Rinse alone
put in quite a number of nights to get things going with these three methods.
In the meantime, I was busy with solving all the integration problems that naturally showed up (I don't even want
to think what it would have meant to delay the integration for the last week, as we were considering at some point).
I have mentioned all these integration problems in my personal log, so I will not repeat them here. Unfortunately,
even though we did manage to solve most issues, there are still a few which will probably break things when least
Gideon has been working a lot on finishing up most of our test cases, including the comparison test with last
year's storage. By doing this, he discovered (and fixed) many tricky bugs and differences, which would have otherwise
gone unnoticed. After doing this, he took up the task of refactoring (or at least starting the refactoring process)
most of our code, since we all agree that in the way it is right now, it is virtually unmaintainable. However, most
of the refactoring work will probably have to be done next year, since it is a huge task.
I have (as requested) also started working on the documentation
of our work, so that next year's team can pick up
where we leave off. This is not entirely complete at this point, but I will make sure not to leave it like that (even
if it is possible that I will not have it done by Friday, November 11th). We have also discussed many issues (mostly
during the preparation of our presentation) with the current code, which I will try to include in the issues section
of the documentation, but there is a chance some will slip my mind. We could also make all these as issues in JIRA,
but I believe it is better if they are read in the more general context of the documentation, so, unless otherwise
instructed, we will leave them out of JIRA for now.
- 09 Nov 2005
Today our team has had a meeting to talk about how our work has to be orchestrated, our working points and next deadlines.
We decided that next monday we will merge our work with the trunk. Before that we want to finish a few parts:
- Moving and copying files (sjoerd), this is almost done
- Finish set/get properties (adriano), almost done as well
- Implement getFileDiff (rinse), left completely but seems to be easy
- Improve the way the pathtree is constructed (gideon)
- Remove duplicate registration of the revision number (it's now both in pathtree and in SVNRAStorage) (bogdan)
- Fix committing files, it now sends the entire file as a delta but it should send only the actual delta (bogdan)
Since implementation has been harder than we originally expected (read it took us longer), we decided not to add any more features after Monday, but just fix bugs we encounter and thoroughly test the system. In other words, we will concentrate on the quality of the delivered code.
After that, we will focus on:
- Making sure all our code is thoroughly documented (it already is pretty much, but just a review of the code)
- Improving the (coverage of the) unit tests using the information from the code coverage
- Finishing the functional tests that compare the new implementation with the old one
- Cleaning up the interface. The transaction related stuff is not really needed, javadoc documentation can be improved (using experience from functional tests)
Generally we want to improve our software engineering practices by:
- actually using jira for bugs, missing features and improvements. This will also help others that take over our work
- keeping a more accurate and up to date log (this should improve communication as well)
- tests, tests, tests
- 20 Oct 2005
- 20 Oct 2005
We should mark a rather major change today, since we've switched from the
linear structure we've been using so far to manage local changes (which was
actually just a list of some objects representing changes) to a tree-like
structure, which is supposed to work as a "virtual working copy". We call it
virtual because it actually just mirrors the structure of the repository, but
not the contents of the files themselves. The only contents it actually stores
is the locally changed contents (modified files or properties and the like).
This structure actually hides behind a more (internally) usable interface the
complexity of managing the local changes.
On the less good part of things, we have been told that it might be a better
idea to not mirror the entire directory structure, but rather just those parts
which are locally accessed in a sort of lazy checkout manner. Of course, we
see the benefits of such an approach as well, but also the increased complexity
in structure management, which is why we avoid this optimization for now. If
things work completely with our less efficient implementation, we will naturally
try to integrate this optimization.
Another aspect which is not completely thought through yet is that we intend to
transparently update the virtual working copy as long as read-only operations
take place. We would do this for each call to any of the methods in the storage
interface until a local change would be applied. At this point, we intend to
freeze the version to whatever it is when the local change is performed. The
problem is that this continuous update could prove to be quite heavy weight.
However, this could be mitigated by the fact that we don't expect any actual
update most of the time, since people don't really commit changes all that often
(and, of course, if the version on the repository doesn't change, no update is
necessary). It is clear, however, that some optimization of this aspect might
be needed at some point.
- 18 Oct 2005
Some instructions for performing tests:
Starting the svn test server from the command line
$ cd [whereever/]gw/src/java/gw/storage/tests
$ ./svnstart.sh /nix/var/nix/profiles/subversion-java .
Stopping the svn test server from the command line
$ cd [whereever/]gw/src/java/gw/storage/tests
$ ./svnstop.sh /usr
Changing the dump file which is used to initialize svn test server
// start the test svn server (see above)
$ mkdir temp
$ cd temp
// checkout a working copy
$ svn co svn://localhost:10000/ .
$ cd branches
// in the commands below, replace <your-branch> with a name that (more or less)
// points to the method which you're setting up this test directory for (e.g.,
// I've used commit for setting up a test directory to test the the commit method)
$ mkdir <your-branch>
$ cd <your-branch>
// make any changes you want to the branch (but not to other branches!!!)
// go back to the temp dir (command varies depending on where you are)
$ cd (\.\./)*
// let svn know about your changes
$ svn add branches/<your-branch>
// commit your changes
$ svn ci -m "some log message"
// create the dump file (this directly overwrites the old dump file so
// make sure you do this right!)
$ /nix/var/nix/profiles/subversion-java/bin/svnadmin dump /tmp/test-svn-repo/ \
// stop the test svn server (see above)
- 11 Oct 2005
Bogdan and I talked a lot about redesigning the CommitItemContainer. The current implementation has a serious limitation when performing operations on directories recursively. To remedy this, the new CommitItemContainer will consist of a tree structure of all files in the repository where the directories and files are nodes. It would basically be a virtual file system without the actual files but only the changes to them. I think we should create some small design document, hopefully more news tomorrow.
- 11 Oct 2005
Today most of us worked from 9 till 5 on the project. Several improvements have been made:
- Bogdan made it possible to run tests within the nix build system that use a temporary svn server.
- Some functions have been implemented and/or commited (see the work plan).
- Some design decisions have been made (as listed down here by Bogdan).
- 03 Oct 2005
During Monday's discussion with Eelco Dolstra and Eelco Visser, the following things were
- whether to use a HashMap or an ArrayList in order to store the local changes. The main debate is that HashMap would allow us to only store one change / path, whereas an ArrayList could allow any number of changes / path. The argument for using an ArrayList is that administration of changes would be a lot easier. The argument for using a HashMap is that it would be easier to disallow sequences of actions which should not happen (e.g. add a file and then make a copy on it; this should not be allowed because the repository cannot cope with it). Eventually, the decision was to use the HashMap, although this will make commits more difficult.
- whether to allow the files & directories changed locally to be based on any number of different revisions or on a fixed revision. The latter version means that as soon as the user starts editing, (s)he would no longer get any changes from the repository until (s)he explicitly updates his/her files. The former version means that the atomicity of the user's view of the system would be lost (since (s)he might be seeing files and directories from different revision together, more or less mixed up, with the possibility of confusion). The decision was to fix the revision number on which the local view is based as soon as the first edit operation is performed.
- also, a decision was made about the way to start and stop the svn server. This will be done in the Ant file, with the server being started before tests are run and stopped afterwards.
- 04 Oct 2005
We discussed the following things in today's team meeting.
- how we can integrate the existing work on temporarily storing changes made to files until we commit them to the repository
- what methods of the Storage interface each of us will start working on (see work plan)
- how we should desing our tests for the code we write. The major problem we are confronted with here is that we need a subversion server with controlled data for writing our tests. We though of several options, but haven't made a decision yet:
- start up a SVN server in the script running the tests, set up a repository with controlled data, run the tests and then shut down the server also in the script. This has the disadvantage of the tests being dependent on eachother (if a test changes something in the repository, it might break some of the other tests) and on the context in which they are run (i.e., the script).
- start up a SVN server and set up a repository with controlled data in each test, run the test, then shut down the server also in the test. This makes tests independent, but has a negative effect on performance.
- use an existing and continuously running SVN server in our tests. This has the same disadvantages as option 1 and aditionally might end up changing some otherwise used data. Of course, this could be avoided by setting a dedicated repository for the tests. Another issue with this approach is that the initial state of the repository would always be different (since we will have to test commits as well).
- 29 Sep 2005
Team meeting today. Most important outcome was the decision to use
as our library for implementing
the storage layer based on direct repository access. The reasons
behind the decision are:
- our discovery (and testing thereof) that JavaSVN supports all of the
- the relative similarity between the JavaSVN library and the C SVN client library, which allows us a certain degree of reuse of last year's code (by the possibility of an almost immediate mapping of C SVN library functions to JavaSVN functions);
- the fact that by using JavaSVN (a pure Java implementation) we avoid introducing portability issues in the GW code.
Apart from this, we have also assigned the following tasks:
- Rinse will translate all Dutch comments from the code to English;
- Bogdan will
- set up the storage subdirectory/package so that separation of storage implementations is made clear
- set up the framework for our own implementation;
- Everybody will study the cache mechanism implemented by last year's team in a separate branch, with the intention of reusing it as much as we can. Each team member was additionally assigned a method or two from the Storage interface to focus on while examining the cache mechanism.
Next team meeting is on Thursday.
- 26 Sep 2005
I've generated and made available the javadoc documentation of
JavaHL. After looking for it for way too long on Google, I eventually
decided to generate it myself. If you also need it, here it is:
JavaHL API Documentation
- 25 Sep 2005
After both Gideon and I had some trouble getting the project to run on our home machines (and had to dig somewhat to get to the bottom of the issues), Gideon set up a nice troubleshooting page
for anybody else who might be in the same situation. Hopefully, somebody will find it useful.
- 22 Sep 2005
First project meeting with all team members
and Eelco Dolstra. Eelco gave a short introduction to last year's code and the inner workings of SVN. During this overview we discussed the options on how to go from here. Either extending the current RA c-library or using the JavaSVN
implementation. So for this week we are going to try to get a more indepth knowledge of the current RA c-library and do some research on the JavaSVN implementation. Next week we should all be familiar with the current Storage Management implementation (including the RA development branch
) and be able to decide how to continue development.
- 19 Sep 2005