[wplug] open source software development methodology

[Patrick Wagstrom]

Alexandros Papadopoulos wrote:
> I assume there are a number of people on this list that have
> first-hand involvement in free software projects, so I'd appreciate
> your experiences, DO's and DONT's on the below scenario.
> 
> A loose team of software developers and web 2.0 architects has been
> working on creating a web application that could be called a portal, a
> wiki, a CMS, a knowledge management system and a social networking
> tool all at once. The base layer is mediawiki, with several add-ons
> and some glue between them. Development of this suite has been
> happening in a quite close circle of people up to now, and no real
> software development methods were put in place.
> 
> Now we're thinking of opening up the suite for everyone to use and
> extend. But we don't know how, from a practical standpoint. We want to
> make it as easy as possible for interested third parties to grab a
> copy of the code and run a local, independent version with minimum
> hassle, without relinquishing control of the production copy of the
> code and the stability of the main sites that sit on it (one of them
> being http://openmethodology.org).
> 
> People with no serious software development experience have come
> together and drafted something like this:
> 
> 1. There is the production server and the development server. Noone
> has direct access to the production server.
> 2. Developers have accounts on the development server, and access to
> some version control system that allows them to checkout and merge the
> latest changes in the code.
> 3. When a developer/contributor believes a change he/she submitted
> needs to go to the main branch (the next version of the production
> codebase), he/she notifies a team of approvers.
> 4. The approvers have access to scripts that allow the newly submitted
> code to be incorporated into the the main development site's suite.
> Thus the code is tested for deficiencies, if it breaks anything etc on
> the development copy of the web suite.
> 5. If anything breaks, the production copy of the codebase is copied
> back from the production server to the development server, thus
> undoing any changes.
> 6. If it looks good on the development server, another magic script is
> executed that copies over everything to the production server.
> 
> Now, I see a lot of holes in this, but please remember it has been
> drafted by people who know next to nothing about software development.
> 
> I'd appreciate your comments/thoughts on the above, and/or pointers to
> development documentation/procedures that you might have used and
> found reasonably effective.

Alex,

The people without the software engineering background may be on to 
something.  In my experience developing web apps, having a broken 
production system is bad news, but it can be really hard to manage and test 
everything. You're also trying to address creating a complete development 
community, which is another question completely.

I'll address my thoughts on keeping the main server safe:

1. Make sure that EVERYTHING is versioned -- including the database.  Yes, 
this requires some creativity on your part, but there are various methods 
for ensuring that this work.  A good to do this is make sure that the 
normal database user does not have modify access on the schema, and then 
version everything.  Jeff Atwood wrote about this a few months ago:

http://www.codinghorror.com/blog/archives/001050.html

2. Don't worry about the nuances of the different version control systems, 
whether or not you should use Git, Hg, Bazaar, SVN, Visual Source Safe, 
Clear Case, or CVS.  Seriously, pick one that your team will use and stick 
with it.  It helps if this has a distinct concept of a trunk and makes 
branching easy.  For our system we use git which means that each user has 
their own trunk.  We also have a "development" and "production" trunk.  We 
cherrypick revisions from development into production.  The production is 
always then updated from the git tree so we can roll back at a moments notice.

3. ReviewBoard is your friend.  Great little piece of software.

4. BuildBot is also your friend.  Most web systems don't have traditional 
buildscripts -- but you can unittest the heck out of them.

5. Selenium is great for testing Web Apps

With regards to allowing other people to get invovled, there is one key 
piece of advice.  Give up control.  Communities that hold their code to 
themselves without giving it loose.  OpenSolaris and OpenOffice are great 
examples of failed communities.  GCC is a similar case in that if you 
submit a patch to the core it's basically never going to get implemented, 
they only allow innovations at the edges.  Eclipse, Apache, and Python do 
great jobs of allowing people to contribute to the core, because they 
realize that they need to give up some of the control.  Want another 
example?  Witness the change in Ubuntu as it has had more rigid process 
installed, vs Fedora which has loosened up.

Anyway, hope some of this helps.

--Patrick