The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.


hi all,
  we have clients ranging from 2 users to 2000 users. from 8-5 each of these users use only our software. and we want to start logging what each user did, starting from log on to log off.

if we go the database route logging everything to tables( logically best to implement), database will be immense.

one way i was thinking of was every 6 months or so, dump those tables and then reuse them.

 any one with better ideas?

Send private email
Saturday, July 23, 2005
Log to a separate disk and run a script to delete the oldest logs. If there are some items you always want to keep then don't delete them. You can also keep summary data in a history log if you don't need to trace back to source data.
son of parnas
Saturday, July 23, 2005
If each customer has their own database, it might be useful to have all of these logs go to a centralized database along with identifiers back to the customer db.

This will let you run reporting and see cross-customer information and detail.  Besides, then it should be easier to run comparisons and see how Module X works with 200 users instead of 10.

Just a thought.
KC Send private email
Saturday, July 23, 2005
1. Log important events.
2. Figure out if action logs needs to be inserted into the master log table RIGHT AWAY. (periodic bulk loads are much faster, at the end of the bulk load rebuild the index, you then rename the obsolete master table for this new updated master table, that way things are always online)
3. Are you Kim Jong Il? Why are you spying on your customers?
Li-fan Chen Send private email
Saturday, July 23, 2005
thanks for the ideas.

we have to log because our clients are susceptible to malpractices and also mandated by the government.

so logs will have to be available for 7 years.

So you can imagine the amount of data that will be collected.

Send private email
Saturday, July 23, 2005
Also to note is that the logging itself will become a performance hit to the system if it is run in the same thread.

We had an app that was logging so much it doubled the length of time it took to opperate (somebody made it open and close the file after each log message BAH! ) and it got even worse when the log file was not on the local PC but on an NT file system where it had to check for permissions ot edit each time.

A database would be much better than a plain file, might actualy be useful.

Sunday, July 24, 2005
Logging always needs to queue immediately to a background thread. If you keep a preallocated circular queue for logging the performance can be almost nothing. I've had very tight performance apps dissapointed that they couldn't blame logging for their performance problems.
son of parnas
Sunday, July 24, 2005
Databases keep a log anyway for redo purposes.
Figure out how to mine the info from that.
Then, if necessary, write more info about users and their actions in your application schema(s).
Abstract Typist Send private email
Monday, July 25, 2005
Abstract Typist has it right.  Use the database journal itself.  Along with snapshots (from your backups), you can 'replay' the log using standard 'recovery' tools.
new nick, new rep
Monday, July 25, 2005
Redo logs are designed for efficiency more than for easy reading. With tools such as Oracle's Log Miner, yes, you can play back transactions and see what happened. But do you really want to keep around umpteen GB of redo logs? Can you picture your DBA's expression when the auditors want to see exactly what happened that day 4 years ago? (Three major releases and a couple of patch sets ago, that is.)

And then when your CEO goes golfing with the Orfomsql rep and decides to move off of Myaclix Server, what happens?
George Jansen Send private email
Monday, July 25, 2005
>  Redo logs are designed for efficiency more than for easy reading.

They also probably don't include the information or the events you wish to log. Someone wants to see an event like deposit made by XX for YY etc, not balance record ID updated to YY.
son of parnas
Monday, July 25, 2005
This is not logging as programmers understand it (debug, trace, info, config ...). Instead this is part of your business logic and is a far from trivial problem. To figure out how big of a problem this is look at the liabilities implied by the business case. The more $$$, the bigger your problem.

You need to address this issue as business requirement not as a system maintenance requirement. Anyway, you need to look at things like:
- log consistency: force programmers to log consistently, based on a given business logic (as opposed to adhoc)
- log security: do you have sensitive data in the logs? do you have to encrypt the logs?
- log persistency & performance: how do you persist all this high througput data without a performance hit on the system? how do you query the logged data?
- log archival: 7 years is a long time. You need to find ways to archive data and purge your OLTP system of dormant log records.

Before you come with a solution, make sure you know what the problem is!
- log analysis and data mining: ... I guess for later.
Dino Send private email
Monday, July 25, 2005
"database will be immense"

Could you quantify "immense" in GB/Business day?
Just me (Sir to you) Send private email
Monday, July 25, 2005
how immense is data?

at the moment we have very crude way of logging and
well for on an average 60 - 80 users, per day in text files, it will be anywhere between 200-350 MB.

we are required to log even if some body views any thing. it seems to be a bit too much, but first biggest step is to log any data that changes.

thank you
dan moore
Monday, July 25, 2005
Assuming you are in the low GB range/day, and you have the long overnight window, what is wrong with copying the logs to your log archival system overnight, and clearing the application side log for reuse?
Just me (Sir to you) Send private email
Tuesday, July 26, 2005

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz