Eric Bouwers Thesis Page

Project

Project title:	Improving Automated Feedback
Advisor:	Johan Jeuring
Start date:	05-03-2007
End date:	27-09-2007
Topic / Area:	Generic programming, Rewrite rules, Educational programs

Abstract

Although research shows that high-quality feedback can be generated in educational tools, the current level of feedback leaves much to desire.
In order to improve the quality of immediate feedback in educational tools, we propose the implementation of a generic framework that can be instantiated for different domains.
By combining the most promising lines of research into a single framework the knowledge about algorithms is centralized, allowing it to be easily accessed by manufacturers of educational tools.

Proposal

You can download the proposal for my thesis here.

Thesis

You can download the thesis itself here.

Code

The code of my thesis can be found in the following repository:

• https://svn.cs.uu.nl:12443/repos/rfg/

Thesis defense

The page of my thesis defense, including an abstract and the slides used, can be found here.

Progress

For the latests thoughts about my thesis you can always look at my blog under the label Thesis.

Week 10

• Initial set-up of the wiki on http://ideas.cs.uu.nl/wiki
• Created (read: shamelessly stolen from Thomas who stole it from Stefan and Andres) a nice layout for the thesis
• Improved layout of the thesis with a nice title-page
• Set up the repository for the project here. (Explanation)
• Small attempt for a logo for the ideas-website

Week 11

• Finished initial structure of the thesis
• Copied parts of the proposal to the thesis
• Studies papers and Thesis of Hennecke (See attachment table)
• Contacted Hennecke with my understanding of his algorithm
• Attended Feedback meeting

Week 12

• Initial design of GUI
  • Drawing of screens
• Initial prototype
  • Generic code relying on library functions
  • Three libraries with specified functions
  • Instantiation of the generic code with different libraries
• Description of GUI design
• Check whether part I is indeed the parts from proposal

Week 13

• Describe overall design
• Read and use:
  • How to write a Haskell program
  • Haskell Hacking
• Semi-formal description of phases one and two
• Finish initial GUI
  • Layout as in drawings, using CSS

Week 14

• Finish initial GUI
  • Interactive feeling, using javascript
  • First interaction with backend
• Describe design GUI
• Rough description of phase three
• Rough description of phase four
• Discuss rough descriptions with Johan

Week 15

• Reread Phase 3 and Phase 4
• Reread Design GUI
• Tried to integrate rounded corners based on CSS, failed horribly
• Start implementation GUI for fractions
• Discuss RFG design with Johan
• Read up on confluence
• Discusses algorithm with math-teacher (aka mom)
• Work out new algorithm on examples

Week 16

• Finish GUI for fractions
• Finish chapter on the design of RFG
• Meet with Johan for discussion
• Feedback meeting
• Read:
  • paper: 'The Ergonomics of Computer Interfaces'
  • paper: 'TGraphics in the Views System'
  • Information about JSON, AJAX andPrototype.
• GUI:
  • Added unit-tests for scripts written by myself

Week 17

• Implement parser for fractions
• Implement parser for equations
• Implement solver for fractions
• Implement generic structure for library
• Implement phase 1
• Implement phase 2
• Implement phase 3

Week 18

• Rule-parser for fractions
• Added command-line options
• Read through Classroom-based Research in Mathematics Education
• Read papers about generic programming in Haskell:
  • Programmable Rewriting Strategies in Haskell
  • Generics for the Masses
  • Type-Safe Functional Strategies
  • A Generic Programming Extension for Haskell
  • A Strafunski Application Letter
  • Scrap Your Boilerplate Reloaded
  • A Generic Recursion Toolbox for Haskell
• Read paper Plugging Haskell in

Week 19

• Apply for fractions (with wheres)
• Feedback Meeting
• Implement rule-calculator for fractions (with proper restrictions)

Week 20

• Implement Diff-algorithm for rules
• Define tests and criteria correctness for rules

Week 21

• Implement guess-rule algorithm for phase 3
  • Implement for single case
  • Implement recursion
• Cleanup code implementation phase 3
• Implement checks for rules
• Make remapping of variables external from distance algorithm
• Implement merging of rules with matching RHS and LHS
• Add names to rules
• Finish the fraction tool to be callable from the command line

Week 22

• Implement command line tool for logic
  • Instances for logic
  • Parser for logic (+ tests)
  • Solver for logic (+ tests)
  • Tests for rules
  • Test scenarios for phase 3
• Add more tests for scenarios of third phase for the fraction tool

Week 23

• Fine tune the third phase
  • Make sure that '+', '*', '/\' and '\/' have a list of children instead of always 2
  • Check the effect of removing Parens from data-types
    Has no effect, makes less tests pass even after adjustements. The parenthesis seem to capture the intentions better and are therefore allowed in the data-type.
  • Rewrite test-framework with reuse, lots of duplication right now
  • Make code third phase more clear, split up definition
  • Think about better matches
    A better match in the rule-generation causes some of the tests to pass because the actual rule is more precise, the noise is cut down to a minimum. This is implemented with an inspection of the commutativity and associativity of an operator.
  • Think about how to deal with the arbitrary 5 in the calculation of the PT-distance, this should be less arbitrary
    The arbitrary 5 is replaced by an parameter that represents the extra costs of the steps. This parameter is increased in every recursion, so steps that are further away are less likely to be the closest match.
• Attend feedback-meeting

Week 24

• Fine tune the third phase
  • Think about how to deal with different sizes of children
  • Look into the merging of the rules for logic
    Solved by adding the condition that rules may never merge with themselves. This is checked by finding the name of the rule within the name of the joining rule.
  • Look into the last failing tests
    Problem that arises is that the precedence in the parser can lead to a less accurate rule. Unfortunately, there is nothing that can be done to prevent this in general.
    Most of the problems can be solved by a better match. This is done by inspecting a constructor whether it has a single child. When this is the case the child is matched against the original other. The extra costs involved is one for the adding of the 'missing' constructor. This solves several problems and introduces only one new problem. In the end it makes the algorithm better.
• Write about third phase (see the blog)
• Implement configurable feedback-message
• Think about implementation of the fourth phase
• Meet with GP team

Week 25

• Write chapter about third phase in Thesis
  • Made a new structure, more organized according to steps in reasoning.
  • Tested whether it makes sense to alter the matching of a leaf and a node in the calculation of a rule, it doesn't.
  • Explain how the algorithm for calculating rewrite rules works.
• Make web-application available for fractions
  • Make the parser of the frontend equal to the parser for the backend
  • Solve problem for none-displayed text-input
  • Solve problem for selecting in graphical view only
    Idea is to generate the representation form the parsed AST within JavaScript? . Then it becomes easier to make links that set the cursor to the right position. Johan and I agreed that this was out of the scope of this Thesis.
• Read paper about Uniplate

Week 26

• Work on algorithm of phase 3 (again), the restriction of 'looking like original PT' does not make sense. Find out following things:
  • Is the problem indeed in the recursion because terms are getting smaller?
    Problem was not in recursion.
  • Are the generated rules accurate enough?
    Actually, the rules are too accurate, the meta-variables in the rules introduce dependencies that might not be there. Therefore, the rules are without meta-variables again.
  • How can we incorporate better matches other then size of the environment, maybe bonus points for getting complete LHS correct?
    Not needed after update of rewrite-rule calculation.
  • Algorithm of term-distance seems oke, does it work correctly in the implementation?
    The algorithm is extended to take into account the properties of operators, just like with the rule-calculation. This gives even better output then before.
    

Conclusion is that the algorithm benefits from being easy, the simplest solution provides the best results.

Week 27

• Test the ICFP contest
• Cleaning up of code
  • Remove dead-code
  • make use of crush :: a -> ([a],[a] -> a) instead of current solution.

Week 28

• Write chapter about third phase in Thesis
  • Explain rule-calculation
  • Explain term-distance
  • Explain General recursion
  • Explain rule-merging
  • Small conclusion
  • Proof-read it

Week 29 - 32

• Attend World Scout Jamboree

Week 33

• Finish coding for GUI
  • Make interface available in dutch
  • Add help-facility (both languages)
  • Add button to go the next exercise
  • Add a thanks page
  • Add actual tool
  • Add way to tell the user he is doing well
  • Test page in different browsers (partly done, have to get JS working in IE)
  • Try to implement Ajax update (4 hours) (to be done when really necessary)
• Finish coding for framework
  • Implement phase 4
    Core functionality working now. The available tests pass.
    • Better apply, takes into account operators
    • Main function and function to apply rules and keep the history
    • Function to update assessment of buggy rules
    • Tests for functions
  • Finish up loose programming ends
    • Move all configuration to separate module
    • Make all message configurable
    • Do some profiling for optimization, seems to not be necessary
  • Generate special output when an exercise is done
  • Add option to give feedback in other language
  • Implement equations
    • Finish parser
• Incorporate initial feedback from early tester(s)
  • Adjusted third-phase message
  • Fixed GUI problems
• Process design up until section 5.4
• Start finding testers (sending mail and writing about the web application)

Week 34

• Initiate and process more test results (fractions + logic)
• Write implementation section
  • GUI part
  • RFG part
• Write appendix about rules
• Write appendix about test results (but removed appendix again
• Reread thesis up until chapter 6, adapt where necessary
• Implement equations
  • Finish solver
  • Write tests for phase 3
  • Make GUI for equations
• Ask Jurriaan for feedback on thesis

Week 35

• Initiate and process last test results (equations)
• Finish rough version of Thesis
  • Write abstract
  • Finish evaluation section
  • Write conclusion
  • Write future work section
• Reread complete thesis, adapt where necessary
• Ask Jurriaan for feedback on thesis and adapt

Week 36

• Ask Johan for feedback on thesis and adapt
• Go through code, add comments where neccesary (RFG)
• Start on presentation

Week 37

• Add README with top-level design for GUI
• Re-read README
• Get feedback on thesis from Johan and adapt (Up to chapter 5)
• Work on presentation

Week 38

• Work on presentation
  • Main outline, concrete examples
• Adapt thesis given remarks Johan

Week 39

• Work on presentation
  • Finish all slides
  • Practice
• Give presentation

-- EricBouwers - 28 Sep 2007