Jeanne Hofmans
Students
Name: Jeanne Hofmans
Email:
jdhofman@students.cs.uu.nl
2002-2003
Period 1
Period 2
Period 3
Period 4
Period 5
2003-2004
Period 1
Period 2
Period 3
Period 4
Thesis Project
Topic/Area
Program Verification
Project
Project title: Inventarisatie testmethodieken voor Transfer Solutions bv
Advisors: Doaitse Swierstra, Albert Leenders
Start date: 01-06-2004
End date: 31-12
Description
Planning
- -06/02-07 (23/27) Plaatsen testmethodieken in V-model.
- -07/23-07 (28/30) Analyse testtechnieken.
- -07/06-08 (31/32) Vakantie.
- -08/03-09 (33/36) Analyse strategie\"{e}n Transfer solutions (project Sociale Dienst).
- -09/08-10 (37/41) Teststrategie voor verschillende ontwikkelomgevingen.
- -10/05-11 (42/45) Teststrategie voor verschillende architecturen.
- -11/03-12 (46/49) Toepassen onderzoeksresultaten op situatie bij Transfer Solutions.
- -12/31-12 (50/53) Afrondingsfase.
Thesis
* Week 23
After finishing my thesis proposal I started setting up Latex and Bibtex. I use Bibtex to make a glossary which is nice for the people who will later read/use my work. In this Glossay I describe all kinds of terms like verification, system, etc.
Furthermore I did some reading: Testen op basis van kwaliteitsbehoeften van gebruikers, Developing a new Testing Model andChecklist gestructureerd testen. Especially the first article was very interesting. I will use it in the second part of my research when I define concrete testingtechniques.
* Week 24
This week I looked through MIL-STD-498, RUP and CDM again to see if I could organise the development proces in phases that would be usefull for my project. CDM Classic has many activities that overlap several phases and is basically datadriven. (It is especially developed for and by Oracle.) RUP has an evolutionary approach and is therefor very riskdriven. Since the topic of my research is testing and testing is all about increasing the quality of the software RUP provides some usefull ideas. The problem however is that it is more complex to understand and not easy to put in the V-model. This all made my mind a bit blurry... So I started writing several activities of CDM on separate pieces of paper.
* Week 25
In this week I continued with my separate-pieces-of-paper-approach but now for the activities in MIL. This helped me clear my mind and after a meeting with Albert I was ready for the real work. I knew which fases would be the basis for my research.
So I started writing down the phases and for every phase the activities and products that belonged to that phase. Whenever possible I tried to make comments about testing activities. I based these mostly on MIl-STD-498 and CDM.
Then on Thursday I became 24 years old!!! This was cause for a little celebration!
* Week 26
This week began by the general desription of my approach. Here I described how the phases I defined relate to eachother and how to use it with an incremental or evolutionary approach. I tried to use as many ideas of RUP as possible. The rest of the week I spent finetuning the phases and their activities and products. Every phase now has a series of testing activities that should be conducted in that phase. I just name them and put a reference to the sections where they will be described in depth. At least in the future (as planned). For now I mainly have the section headers. The main sections are: Testondersteuning (planning, specification), Verificatie (unit, intergration en systemtests) en Validatie (requirements, design, system).
* Week 27
On Monday I had a talk with my two advisors. Their main comment was that my paper was already quite large and it was hard to get some oversight. So this week I spent some time shaping my paper; writing introductions and conclusions for several sections explaining how they ralte to my final goal.
Then I spent some time in incorperating RUP into my approach. This didn't take much time since most features were already incorperated into my paper. New things were the iterative development approach, the focus on risk analysis and the introduction of milestones. It came to my attention that activities that I had placed under Quality Assurance were part of the normal proces of RUP.
On thursday and Friday I spent my time on making a table with all phases of the three used standards. Here I noticed that MIl-STD-498 did not really have phases, but just described activities. This is known as an open development method. This caused some rework of earlier sections and the table.
* Week 28
This week I made some adjustments to my paper. The rest of the week I mostly spent on reading Software Testing in the Real World. Here I ran into a little contradiction between MIL-STD-498 (V-model) and the Dotted-U-Model. The way they interpret validation and verification and especially the way they place it in the proces is contradictionary. Where MIL-STD-498 validates the requirements, they are verified in the Dotted-U-Model. This made me decide not to differentiate between validation and verification when discussing testing activities, but to make a distinction between reviews and tests.
I also read something about the SEI Capability Maturity Model (CMM). In kew about htis model before but decided not to use it, because it was to abstract for my goals. Now I decided it would be a nice to refer to and to use it as some kind of metric to see if the proces I defined conforms to this standard. Then I found out that a new model was recently presented by SEI: the Capability Maturity Model Integrated. The main difference is that the new model focusses on iterative development. This indicates I have done something right in defining the development- and testing phases.
* Week 29
The first two days I spent on static testing. Static testing (reviewing) is used to manually verify or validate specifications, design and code (without running it). Reviews are in the form of inspections, walk throughs and buddy checks. Formal inspections have over the years proven to be the most effective way to discover errors. For efficient testing one should use them to at least verify the specification and preferably also the design. Most errors (70 %) occur during specification and design. Another benefit of reviews is that it encourages communication.
The last three days I unfortunately had the flu. I read something in TMap, but I was not able to do more.
* Week 30
Dynamic testing, testing (validation or verification) by actually executing the program, was the subject of this week. Unit, integration and system testing are three major forms of dynamic testing. They indicate the scope.
Besides dynamic and static testing there are various supporting activities: specification, design, management, etc. I identified and described these also.
Finally I spent some time on metrics, tools and techniques, where I focussed on metrics. Measuring = knowing is the key principle of collecting and analyzing metrics. There are sseparate metrics for the proces and product.
* Week 31 and 32
Vacation!! I painted my house and now it looks beautiful.
* Week 33
This week I was supposed to start on analyzing the situation at Transfer Solutions. However I had misjudged the time I would need for analyzing testing techniques. Therefore I changed my planning a little bit.
Thus I spent two days on discussing how to define the proces. This means I compared methodologies, development strategies and models.
* Week 34
Thinking about how to do my practical research at Transfer Solutions... They gave me a project to evaluate. I started to look at the data I collected, but I soon realized it was not my task to analyze the data itself, but to analyze what hte collected and why so I can later give tailored advice. Thinking about this made me reformulate my research question:
Welke maatregelen moet men kijkend naar kosten (tijd en geld) en baten (kwaliteit) treffen om het testproces als geheel, maar ook per fase in het V-model, zo efficiënt mogelijk in te richten?
After that I started on the list of questions I want to ask. These all have to help me reach my goal. My advisor Albert Leenders helped me to keep that in mind.
Furthermore I had an interesting talk with another collegue about testing. Some things I can use in the future, other things were about my current report. She asked a very interesting question:
How do you know after an increment wich systemtests have to be run again? The same goes for a bug fix. To rerun the unit tests is usually easy since they are automated. Most systemtest are not. Tracebility was my answer, but in practice this becomes very hard. Especially with more complex projects one easily loses oversight.
* Week 36
Most of my time I spent on making up my questionaire. Next week I will interview the various teammembers of the project of Sociale Dienst Amsterdam. The questionaire has several sections most of which are similar to my report. Although it is quite extensive (16 pages), I reckon that during the interviews I wil come up with extra questions. Maybe I incorperate these in my questionaire later. I do feel the questions I have formulated this far form a good basis.
Besides the questionare I worked on a introductionairy chapter called Testproces and on my presentation.
* Week 37
This week I finally had the interviews with the people who work for the Sociale Dienst Amsterdam project. It was very nice to see the differences between theory and practice. Especially because the project has a very pragmatic approach. Nobody really knew what a formal inspection was or even what iterative programming is. Before one might think it is a badly arranged project.. They very consciously register all errors together with their status, type, version, when, etc. I haven't finished working this out. This will be my job for next week.
I also discussed with the projectleader to try out having a formal inspection. He promised me to wait until I have my final advice ready.
The difficulty here will be to keep the pragmatic approach in mind. In other words: my advice should be workable.
* Week 38
I evaluated the results of the interview and started formulating my advice. This proved to be difficult, because real people work on the project. I discussed this with an experienced collegue who has worked as a reviewer for some years. He gave me some hints on how to formulate things and reassured me that they also know the testproces is not perfect. I was not only looking at giving an advice for improving the testproces I also took up all the difficulties the projectleader would face implementing it. The project is however not my responsability, the testproces is.
Besides this I read some more on inspections, Quality in ICT-projects and made some improvements concerning these topics. I also started thinking on which questions I needed to ask on (not) testing generated code.
Furthermore I was wondering who in the world reads this. Although I believe it is obligated I do not think most other students do it. If one happens to read this, mail me.
* Week 39
The Main event this week was Oracle Open World, a promotional event on Oracle products and a way to meet people. I got the phonenumber of the manager R&D Testing at Sogeti. Sogeti is a company that is specialised in testing and the publisher of TMap. Soon I will call to ask some questions.
I also mailed one of the executives of Oracle to find out a detail of his presentation. For the release of Oracle application Server 10g they have spent 2614 man-hours on development and 1072 on testing. Besides that they have spent 3.1 million hours on automated testing. He offered me to sent more information at my request. I was very pleased with that.
Furthermore I worked on testing 4 GL applications.
This weekend I went with the company to Amaland for wadlopen. Besides that we went to the pub and I had enormous fun!
* Week 39
I spoke some more people on testing 4GL and classic or web-based client server architectures. Besides that I promoted my presentation of October 28 by making a beautifull poster.
Literature
Here I present the most relevant literure of my research.
These are the main methodologies I use for reference.
* MIL-STD-498, Militairy Standard Software Development and Documentation. USA, 05-12-1994
* Oracle Method Custom Development, A quick tour of CDM, Oracle 1996
Oracle Method Custom Development, Method Handbook, Oracle 1996
Oracle Method Custom Development, Deliverable reference, Oracle 1996
* Rational Unified Process, Documentation
These books focus on testing.
* Software Testing in the Real World, improving the proces. Edward Kit, 1999
* Testen volgens TMAP. Pol, Teunissen, van Veenendaal. 2000
