The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Forms, Forms, Forms - How to deal with soo many forms?

I have been assigned a new project which is basically to design a web application that can output/fill pdf forms. Currently the company fills hundreds of forms by hand and thus seeks a solution to minimize some of the work load. In many cases there are sets or groups of 20 or so forms whose data is approx 70% in common. In this scenario I can just create one common interface to manage the data and fill each form 70% complete. That in itself will be a massive improvement in efficiency. But, what about the other 30% that is form specific? We need a way of storing that information digitally because it will be reused.

The obvious solution is to just build interface components and database structures for the specific fields on each of the forms we use. But I can’t help but feel there is a better solution to the problem. Ideas?

Lotek Send private email
Wednesday, November 22, 2006
Ideally, there'd be some up-front analysis about why we fill out all these forms, where do they go, what's the data used for, do we really need them all, etc.

Automating a bad process is usually less effective than improving the process.
Mike S Send private email
Wednesday, November 22, 2006
We performed this analysis and unfortunatly couldn't forsee anyway around it. The format of many of the forms are established by private companies or state governments who will not accept any form submissions in anything but "their" form....There is no standardization, which is the main problem...
Lotek Send private email
Wednesday, November 22, 2006
OK, then I would take all the data you're capturing and create a normalized database design for it and store it there.  At least then it will be usable for other purposes in the future.  And then, as you say, populate the forms without the user having to re-enter the same fields.
Mike S Send private email
Wednesday, November 22, 2006
We just had a discussion on this here....

I was looking at the new forms server from MS...

The above will at least give you some ideas....

Albert D. Kallal
Edmonton, Alberta Canada
Albert D. Kallal Send private email
Friday, November 24, 2006
>hundreds of forms...
>sets or groups of 20 or so common.
>In this scenario I can just create one common interface to manage the data and fill each form 70% complete.

If you have many arbitrary forms, then you need to manage the metadata *very* well.

If this is merely an internal application to make life a little easier for your co-workers, then I'd start with a worksheet per form family/group that has the common information (and it may already have some info prefilled, like your company's name and address). Getting a 70% solution should take about 1-2 weeks of effort, and the last 30% will be harder.

If this is an application for sale, or that customers might be using...

Depending on the route you chose, this can go from 12 man-month to 240+ man-month size project (for a commercial app). If you only have 1-2 forms, this could be a 2 week project, if you have hundreds of arbitrary forms, then plan for a multi year project.

Some people will try to define a table per form, which, based on prior experience in this area, will guarantee trouble/grief in the future, as producers of forms will add/remove/rename fields in the future. Table-per-form is appropriate if you have a small number of forms (like 1 or 2) and quickly hits a wall before you hit 10 forms. Entity-Value-Attribute does a better job of maintaining data for the form-fields, but requires metadata to connect form-fields with the forms. Many people despise EAV, which is fine, it just happens that EAV is a better fit for this sort of problem. Some of the forms (used in current and past employers) contain over 1k fields, and I'm not sure if SQL Server or Oracle can handle tables with 1k+ columns.

You may bump into some requirements of the e-sign act which may require you to be able to reproduce older forms (up to 10 years old). So versioning of the forms is something you should plan from the beginning. If you serialize the data that you store, then handling different versions will be troublesome.

I will use an example from a previous employer (a commercial vertical application): real estate forms that go on behind the scenes when someone is looking to buy/refinance real estate. Their primary competition at that time was FedExing of the paper forms. Most of these forms contain a number of common elements, such as borrower's name, address/location of property to be mortgaged/refied. Since the forms are part of a package (a business rule), then having one other form that contains common information (call it an information worksheet if you will) can be part of the workflow. Some of the forms would have part filled out by one party, handed to the next party for more to be filled out, then handed off to yet another party for more data. In this case, a state machine was a critical component, as well as being able to retrieve/handle if one of the later parties changed data that an earlier party entered (i.e. who changed that field, when, and what was the previous data).

It might be that InfoPath or WWF might do this quite easily. I won't have time to experiment with either until early-mid next year. Although if someone from MS wants to send me an NFR copy, I'd get started sooner.

I've been thinking about writing a book about doing stuff like this, maybe I should get started.

UPDATE from previous thread: the interactive (you can fill it out) form 1003 can be found here:
Peter Send private email
Sunday, November 26, 2006
it's same like simulating a protocol.
u have lots of protocols - or actualy a big one consisting of all the subprotocols.

just imagine some old network bit-wise protocol, e.g. banking, whatever. it has hundreds of possible records/forms, with thousands of variations. Now, u have to simulate it...

1. create a formalised "meta-language" to describe your forms/records. Does not mean actual language as operators/syntax, it means language as metaphores and notions

2. create an interpreter for your language - or better, generative parser/interpreter, accepting multiple backends, that is coders/translators. One which if given a proper translator, will translate the metadata of your form into whatever shape it does - be it htmlized form, C source, binary record, printable structure, SQL table....

3. create these translators, whatever u need

4. create some storage for the resulting metadata (form description) - and then for the filled forms

5. educate some users how to use your language and let them create/describe the forms themselves

6. create a (http) server through which the forms, once described, can be visualised/edited/searched

does this sound as multimillion $ 10 year project?
does not have to be.

u can use self-awareness in your core objects - and voila, u have _the_ interpreter.
u _declare_ the objects/structs with all the metadata required, and then the core object/types should  do the job for you - as far as u have written it how to do it.

i wrote a 350-records protocol simulator for 3 months (+http full MVC part). Another guy helped with the generator of the descriptions (out of the protocol's C header files).
so that makes, say, 5 man-months.

later, i used same underlying library for wrapping up some corba-protocol, and generating c++ for it; then about DB-tables/schemas; now i use it for MVC model which generates the View and Controller and the DB-representation out of self.

i was going to publish it, never had the time.
if u're interested, AND able to read such _very_-tight things, i can mail it. or publish it.

as notions, it's about static types and self-awareness, and the idea the (syntax)language for the form is actualy python - but with changed semantics;

as implementaiton, it's all python, metaclasses, and descriptors. Once, declared, the static-typed objects behave like other objects but carry all the metadata u have programmed into it - and hence u can do lots of stuff with it _automaticaly_.

example of usage:

from alltypes import Struct, Number, String

class Person( Struct):
  name = String( )
  age  = Number( typ=int, default=5, min= 0, measure='years' )

class A1(Struct):
  who = Person.StaticType()
  why = String( default='no reason' )
where Struct is the static-Structer (structure), Number,String are static-types (elements).

of course u can use another implementation for such thing, there are several. Look up the all-in-one web-MVC (pylon/turbogears/..); look forms validators, and content-management, e.g.

maybe use whole Zope? although that IS overkill in most cases...

whatever, make sure to have fun.
svd Send private email
Monday, November 27, 2006
You may want to try Wufoo (

Radha Mukkai Send private email
Monday, November 27, 2006

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz