SProject Log 06
Gw
Tuesday 15/11/2005
I have completed the part of the documentation which I was planning to write. -- BogdanDumitriu - 15 Nov 2005After 20/10/2005
We have mostly neglected our log during the past two weeks, so I'm going to try to summarize what our team has been working on lately, for reference purposes. The Monday after the meeting metioned in the previous entry we have indeed proceeded with the integration of our code in the trunk. At the same time, we've found out that Sjoerd was no longer following the Swe course, so Rinse took over his responsabilities, which meant implementing the move, copy and revert methods (the only things that were still not supported by our storage layer). Soon after Rinse started working on these methods, it turned out that we were quite optimistic when thinking that writing copy, move and revert shouldn't be much of a problem. It actually turned out to be quite a difficult task, both in itself and in that a lot of our code stopped working properly with the configurations of our virtual working copy introduced by executing copy, move or revert. Rinse and I have probably spent somewhere around 10 hours (cumulated) discussing various issues that kept coming up, and Rinse alone put in quite a number of nights to get things going with these three methods. In the meantime, I was busy with solving all the integration problems that naturally showed up (I don't even want to think what it would have meant to delay the integration for the last week, as we were considering at some point). I have mentioned all these integration problems in my personal log, so I will not repeat them here. Unfortunately, even though we did manage to solve most issues, there are still a few which will probably break things when least expected. Gideon has been working a lot on finishing up most of our test cases, including the comparison test with last year's storage. By doing this, he discovered (and fixed) many tricky bugs and differences, which would have otherwise gone unnoticed. After doing this, he took up the task of refactoring (or at least starting the refactoring process) most of our code, since we all agree that in the way it is right now, it is virtually unmaintainable. However, most of the refactoring work will probably have to be done next year, since it is a huge task. I have (as requested) also started working on the documentation of our work, so that next year's team can pick up where we leave off. This is not entirely complete at this point, but I will make sure not to leave it like that (even if it is possible that I will not have it done by Friday, November 11th). We have also discussed many issues (mostly during the preparation of our presentation) with the current code, which I will try to include in the issues section of the documentation, but there is a chance some will slip my mind. We could also make all these as issues in JIRA, but I believe it is better if they are read in the more general context of the documentation, so, unless otherwise instructed, we will leave them out of JIRA for now. -- BogdanDumitriu - 09 Nov 2005Thursday 20/10/2005
Today our team has had a meeting to talk about how our work has to be orchestrated, our working points and next deadlines. We decided that next monday we will merge our work with the trunk. Before that we want to finish a few parts:- Moving and copying files (sjoerd), this is almost done
- Finish set/get properties (adriano), almost done as well
- Implement getFileDiff (rinse), left completely but seems to be easy
- Improve the way the pathtree is constructed (gideon)
- Remove duplicate registration of the revision number (it's now both in pathtree and in SVNRAStorage) (bogdan)
- Fix committing files, it now sends the entire file as a delta but it should send only the actual delta (bogdan)
- Making sure all our code is thoroughly documented (it already is pretty much, but just a review of the code)
- Improving the (coverage of the) unit tests using the information from the code coverage
- Finishing the functional tests that compare the new implementation with the old one
- Cleaning up the interface. The transaction related stuff is not really needed, javadoc documentation can be improved (using experience from functional tests)
- actually using jira for bugs, missing features and improvements. This will also help others that take over our work
- keeping a more accurate and up to date log (this should improve communication as well)
- tests, tests, tests
-- GideonSmeding - 20 Oct 2005
Monday 17/10/2005
We should mark a rather major change today, since we've switched from the linear structure we've been using so far to manage local changes (which was actually just a list of some objects representing changes) to a tree-like structure, which is supposed to work as a "virtual working copy". We call it virtual because it actually just mirrors the structure of the repository, but not the contents of the files themselves. The only contents it actually stores is the locally changed contents (modified files or properties and the like). This structure actually hides behind a more (internally) usable interface the complexity of managing the local changes. On the less good part of things, we have been told that it might be a better idea to not mirror the entire directory structure, but rather just those parts which are locally accessed in a sort of lazy checkout manner. Of course, we see the benefits of such an approach as well, but also the increased complexity in structure management, which is why we avoid this optimization for now. If things work completely with our less efficient implementation, we will naturally try to integrate this optimization. Another aspect which is not completely thought through yet is that we intend to transparently update the virtual working copy as long as read-only operations take place. We would do this for each call to any of the methods in the storage interface until a local change would be applied. At this point, we intend to freeze the version to whatever it is when the local change is performed. The problem is that this continuous update could prove to be quite heavy weight. However, this could be mitigated by the fact that we don't expect any actual update most of the time, since people don't really commit changes all that often (and, of course, if the version on the repository doesn't change, no update is necessary). It is clear, however, that some optimization of this aspect might be needed at some point. -- BogdanDumitriu - 18 Oct 2005Tuesday 11/10/2005
Some instructions for performing tests:Starting the svn test server from the command line
$ cd [whereever/]gw/src/java/gw/storage/tests $ ./svnstart.sh /nix/var/nix/profiles/subversion-java .
Stopping the svn test server from the command line
$ cd [whereever/]gw/src/java/gw/storage/tests $ ./svnstop.sh /usr
Changing the dump file which is used to initialize svn test server
// start the test svn server (see above)
$ mkdir temp
$ cd temp
// checkout a working copy
$ svn co svn://localhost:10000/ .
$ cd branches
// in the commands below, replace <your-branch> with a name that (more or less)
// points to the method which you're setting up this test directory for (e.g.,
// I've used commit for setting up a test directory to test the the commit method)
$ mkdir <your-branch>
$ cd <your-branch>
// make any changes you want to the branch (but not to other branches!!!)
// go back to the temp dir (command varies depending on where you are)
$ cd (\.\./)*
// let svn know about your changes
$ svn add branches/<your-branch>
// commit your changes
$ svn ci -m "some log message"
// create the dump file (this directly overwrites the old dump file so
// make sure you do this right!)
$ /nix/var/nix/profiles/subversion-java/bin/svnadmin dump /tmp/test-svn-repo/ \
> [whereever/]gw/src/java/gw/storage/tests/dump-file
// stop the test svn server (see above)
-- BogdanDumitriu - 11 Oct 2005
Bogdan and I talked a lot about redesigning the CommitItemContainer. The current implementation has a serious limitation when performing operations on directories recursively. To remedy this, the new CommitItemContainer will consist of a tree structure of all files in the repository where the directories and files are nodes. It would basically be a virtual file system without the actual files but only the changes to them. I think we should create some small design document, hopefully more news tomorrow.
-- GideonSmeding - 11 Oct 2005
Monday 03/10/2005
Today most of us worked from 9 till 5 on the project. Several improvements have been made:- Bogdan made it possible to run tests within the nix build system that use a temporary svn server.
- Some functions have been implemented and/or commited (see the work plan).
- Some design decisions have been made (as listed down here by Bogdan).
- whether to use a HashMap or an ArrayList in order to store the local changes. The main debate is that HashMap would allow us to only store one change / path, whereas an ArrayList could allow any number of changes / path. The argument for using an ArrayList is that administration of changes would be a lot easier. The argument for using a HashMap is that it would be easier to disallow sequences of actions which should not happen (e.g. add a file and then make a copy on it; this should not be allowed because the repository cannot cope with it). Eventually, the decision was to use the HashMap, although this will make commits more difficult.
- whether to allow the files & directories changed locally to be based on any number of different revisions or on a fixed revision. The latter version means that as soon as the user starts editing, (s)he would no longer get any changes from the repository until (s)he explicitly updates his/her files. The former version means that the atomicity of the user's view of the system would be lost (since (s)he might be seeing files and directories from different revision together, more or less mixed up, with the possibility of confusion). The decision was to fix the revision number on which the local view is based as soon as the first edit operation is performed.
- also, a decision was made about the way to start and stop the svn server. This will be done in the Ant file, with the server being started before tests are run and stopped afterwards.
Thursday 29/09/2005
We discussed the following things in today's team meeting.- how we can integrate the existing work on temporarily storing changes made to files until we commit them to the repository
- what methods of the Storage interface each of us will start working on (see work plan)
- how we should desing our tests for the code we write. The major problem we are confronted with here is that we need a subversion server with controlled data for writing our tests. We though of several options, but haven't made a decision yet:
- start up a SVN server in the script running the tests, set up a repository with controlled data, run the tests and then shut down the server also in the script. This has the disadvantage of the tests being dependent on eachother (if a test changes something in the repository, it might break some of the other tests) and on the context in which they are run (i.e., the script).
- start up a SVN server and set up a repository with controlled data in each test, run the test, then shut down the server also in the test. This makes tests independent, but has a negative effect on performance.
- use an existing and continuously running SVN server in our tests. This has the same disadvantages as option 1 and aditionally might end up changing some otherwise used data. Of course, this could be avoided by setting a dedicated repository for the tests. Another issue with this approach is that the initial state of the repository would always be different (since we will have to test commits as well).
Monday 26/09/2005
Team meeting today. Most important outcome was the decision to use JavaSVN as our library for implementing the storage layer based on direct repository access. The reasons behind the decision are:- our discovery (and testing thereof) that JavaSVN supports all of the
svn,svn+ssh,httpandhttpsprotocols; - the relative similarity between the JavaSVN library and the C SVN client library, which allows us a certain degree of reuse of last year's code (by the possibility of an almost immediate mapping of C SVN library functions to JavaSVN functions);
- the fact that by using JavaSVN (a pure Java implementation) we avoid introducing portability issues in the GW code.
- Rinse will translate all Dutch comments from the code to English;
- Bogdan will
- set up the storage subdirectory/package so that separation of storage implementations is made clear
- set up the framework for our own implementation;
- Everybody will study the cache mechanism implemented by last year's team in a separate branch, with the intention of reusing it as much as we can. Each team member was additionally assigned a method or two from the Storage interface to focus on while examining the cache mechanism.