The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Large mortgage portal suggestions

Hi

I am to develop a real estate related website which will disseminate information to mortgage agents on a daily basis, alerting them of various conditions etc. , based on a database that is populated from daily government supplied information ( This info is not free ).

The expected user base is about 15 to 20000. The volume of data input daily can be a maximum of say 5000 records. However the data is useful only for a certain period and ought to be archived periodically and removed from the main databses.

The website will provide basic member management, charting and a way to subscribe to alerts and so on.

There will also be a desktop client app that will provide a subset of the features of the websites interface ( perhaps a vista gadget or tray application ).

The current exisiting website has been running from 4 years (Win2K server with MSSQL and ASP.NET based website, running on 3 boxes) and seems to be straining under the load. In future the amount of data would increase by a factor of 4 to 5.

I have experience with desktop applications (mostly graphics, multimedia, components and system utilities) with C++ Builder and Delphi ( not to mention a lot of low level C++ code in Visual C++ ). I have no website development experience. However, the sit itself is quite simplistic and I believe I dont need to be so much of a web guru to get this done.

The front end client application would be in Delphi.

I am leaning towards a Delphi for PHP based backend running on a LAMP, perhaps Postgres SQL instead of MySQL.

Maybe Ruby on Rails is better? Codegear has a product called 3rd rail, which seems to be interesting.
Maybe python?

Any suggestions, advice etc. are welcome.
Vivek N Send private email
Tuesday, May 20, 2008
 
 
I hate to say "not enough information" when you've obviously gone to some lengths to make that description useful, but there is still not enough information.

Websites do not typically have performance problems because there are too many users in total.  Rather, the more important issue is peak concurrency.  If your userbase is 20,000, that could correspond from to anything from a peak concurrency of single digits to 20,000 users hitting the server at once at 9:01 on Monday morning. 

Similarly, I'd choose Rails by preference over other web frameworks because I like working in it, but knowing exactly what kind of performance challenges you are looking at would help me firm up that recommendation.  5,000 new records a day is nothing -- thats 18 million records a year, and any scaling issue in that number is solved by really basic things like using indexes appropriately, buying enough hardware for your needs, and otherwise designing well. 

No framework is going to rescue you from poor design decisions. 

As to how easy it is to make the leap from standard client/server programming to web apps, if you can get your head around the MVC pattern (which is the dominant paradigm for web apps), then most web frameworks are not terribly difficult to start working in.  Some are, of course, easier than others -- I do Java at the day job and Rails for my small business, and my experience is that Rails is much more respectful of my time ;)
Patrick McKenzie (Bingo Card Creator) Send private email
Tuesday, May 20, 2008
 
 
>> The current exisiting website has been running from 4 years (Win2K server with MSSQL and ASP.NET based website, running on 3 boxes) and seems to be straining under the load.<<

Resist the urge to rewrite for a while. You need to profile the app to see exactly where the performance/throughput speed bumps are. You might find, for example, that tweaking 1 line of code or 1 configuration line doubles your throughput.

ASP.NET is in general a *very* scalable platform, especially when combined with SQL Server, Oracle, or Sybase database.
Mark Pearce Send private email
Tuesday, May 20, 2008
 
 
Thanks,

Indeed peak concurrency would be an issue, most of the users would want to know the interest rates set by the government bodies each day at the earliest, so the scenario of a large number of users hitting the website at 9:01 Monday ( and indeed every day ) morning is very likely.

I know that as a language, PHP is not anywhere as advanced as Ruby or Python, but my in my brief lookover of Delphi for PHP, I notice it uses the Delphi VCL framework, ported over to PHP. This would give me a headstart as I am already familiar with the VCL and practically all i have to do is mentally translate between Object pascal and PHP as I write the code, which is nowhere as hard as learning a new framework or paradigm from scratch.

Does anyone have any views on the 3rd Rail product?
Vivek N Send private email
Tuesday, May 20, 2008
 
 
Having run database-heavy sites on Win2K, ASP, SQL, there is no reason that (appropriately provisioned and developed) it cannot handle the load - I suspect the architecture, design or construction of the site is the problem, especially given that it sounds like you already have separate SQL Server and IIS.

Millions of rows in many tables, multi-GB tables in many-GB databases, properly indexed, not a problem in MSSQL, and I don't even bother archiving.

To be honest, I would stick with the same architecture and re-use as much as possible on Win2008, ASP.NET, SQL2K5.

Since the desktop front end is a subset, and even if it needed a certain level of interactive richness, I would also make it a web-application on the same server - if richness exceeded my pain threshold for pure web, I would think about WPF/Silverlight/AIR or similar connecting to web services exposed by your core web app.  I wouldn't bother with yet another separate desktop application unless there were very specific indications of functionality which were best handled that way - saves an installation and support.
Cade Roux Send private email
Tuesday, May 20, 2008
 
 
And 1.8 million records a year is not a lot - I have one ETL package which is bringing in 18 million records a day, with a 6-month retention window.
Cade Roux Send private email
Tuesday, May 20, 2008
 
 
>I have no website development experience.

this can only end badly. how did you get the job to, you know, develop a website, if you have "no website development experience"?
please don't delete
Tuesday, May 20, 2008
 
 
>The current exisiting website has been running from 4 years (Win2K server with MSSQL and ASP.NET based website, running on 3 boxes) and seems to be straining under the load. In future the amount of data would increase by a factor of 4 to 5.

I agree with what other people say. ASP.NET is a ~very~ scableable framework (MySpace are doing alright with it) and so changing to something else is probably not the answer. I would look for the bottlenecks and fix them in the existing system.
Craig Send private email
Tuesday, May 20, 2008
 
 
Well I am used to tackling brand new technologies or development stuff and coming out on top.

I learned Windows and DirectX , Delphi, C++, Java and network programming simply by starting directly on large projects and plunging ahead, having little or no knowledge of the details involved, picking them up and going on.

All those projects I started in unknown areas did work out fine, so I see no reason why web development should not.

And as far as I can see, this websites complexity is very simple - Manage registered users, display periodic information from a database, send a few e-mails or SMS when certain conditions are triggered.

Even If I had to write the whole website as a monolithic C++ CGI application (not considering seriously, just making a point) , it would still seem doable me, so all the PHP or other web oriented stuff can only make it vastly simpler.

I have written data visualization components before, tyhe only difference here is the output is on a webpage rather than an application.

Again I may go against conventional wisdom by rewriting the system from scratch, but I think that thats a much better excersize than trying to wade through reams of old code which isnt documented and whose developer is dead ( he is ).
Vivek N Send private email
Wednesday, May 21, 2008
 
 
Even if you rewrite, I would rewrite again in C# ASP.NET. Some Linux weenies may dis it, but people who know, know that it rocks (especial with MS MVC).
Craig Send private email
Thursday, May 22, 2008
 
 
"Again I may go against conventional wisdom by rewriting the system from scratch, but I think that thats a much better excersize than trying to wade through reams of old code which isnt documented and whose developer is dead ( he is)."

You're effectively going to do that with your own code anyway.  Since you are new to web development, you are going to have to make a lot of mistakes and learn best practices/style and refactor/rewrite for whatever web platform you choose; i.e. your first attempt may be likely to have just as many issues.
Cade Roux Send private email
Thursday, May 22, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz