[wplug] open source software development methodology

[Bill Moran]

In response to "Alexandros Papadopoulos" <apapadop at alumni.cmu.edu>:

> I assume there are a number of people on this list that have
> first-hand involvement in free software projects, so I'd appreciate
> your experiences, DO's and DONT's on the below scenario.
> 
> A loose team of software developers and web 2.0 architects

Why not web 3.0?  Web 2.0 is Soooo 1990s!

> has been
> working on creating a web application that could be called a portal, a
> wiki, a CMS, a knowledge management system and a social networking
> tool all at once. The base layer is mediawiki,

There's your first mistake ...

> with several add-ons
> and some glue between them. Development of this suite has been
> happening in a quite close circle of people up to now, and no real
> software development methods were put in place.
> 
> Now we're thinking of opening up the suite for everyone to use and
> extend. But we don't know how, from a practical standpoint. We want to
> make it as easy as possible for interested third parties to grab a
> copy of the code and run a local, independent version with minimum
> hassle, without relinquishing control of the production copy of the
> code and the stability of the main sites that sit on it (one of them
> being http://openmethodology.org).
> 
> People with no serious software development experience have come
> together and drafted something like this:
> 
> 1. There is the production server and the development server. Noone
> has direct access to the production server.
> 2. Developers have accounts on the development server, and access to
> some version control system that allows them to checkout and merge the
> latest changes in the code.
> 3. When a developer/contributor believes a change he/she submitted
> needs to go to the main branch (the next version of the production
> codebase), he/she notifies a team of approvers.
> 4. The approvers have access to scripts that allow the newly submitted
> code to be incorporated into the the main development site's suite.
> Thus the code is tested for deficiencies, if it breaks anything etc on
> the development copy of the web suite.
> 5. If anything breaks, the production copy of the codebase is copied
> back from the production server to the development server, thus
> undoing any changes.
> 6. If it looks good on the development server, another magic script is
> executed that copies over everything to the production server.

This is doomed to fail.  Steps 2-4, in particular, will set up a scenario
where the "development" version differs too much from the "production"
version to merge changes in any more.  What happens to improvements that
don't get approved to pull into the production version?  What happens when
improvements that are approved for the production version are dependent
on previous improvements that weren't approved for the production version?

> Now, I see a lot of holes in this, but please remember it has been
> drafted by people who know next to nothing about software development.

That's pretty obvious.

> I'd appreciate your comments/thoughts on the above, and/or pointers to
> development documentation/procedures that you might have used and
> found reasonably effective.

It seems to me that you're starting off on the wrong foot.  Based on your
description of the situation, it doesn't seem like you currently have a
version control system internally.  That's a mistake that you need to
correct before you go open source with the software.  Get a VC system
(subversion or git or whatever) in use internally and establish a sane
release engineering process, with rules for code review, branching, testing,
and versioning.  Once you have those in place, I think you'll find the
process of taking that existing procedure open source will be pretty
obvious and easy -- it's just a question of whether or not you want to
give people outside your organization commit access or not.

There are different ways to handle checking in outside code, but it will
always involve _some_ sort of review process.  The FreeBSD process is
well documented:
http://www.freebsd.org/doc/en/articles/releng/
Personally, I prefer the way the PostgreSQL project is run over all other
projects I know of:
http://blogs.ittoolbox.com/unix/bsd/archives/josh-berkus-of-postgresql-8710

I suspect that some googling for phrases such as "open source release
engineering" will turn up lots of other processes.  A search on Amazon
or one of their competitors will probably turn up a few good books.

I recommend not inventing the process yourself, but finding an existing
project out there that mimics your project and adopting their way of
doing things.  In my opinion, the key factors are:
*) Having a schedule for releases, including feature freeze and beta cycle
*) Have a clear policy for branching, merging and other VC operations
*) Having a clear process for code review
*) Setting goals for release and rules on what can be committed and
   what doesn't belong in each release

-- 
Bill Moran
http://www.potentialtech.com