The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

HTTP vs FTP for software updates.

I have a nice little FTP set of code in VB  (actually, it VBA). This code uses the wininet.dll…and has a call back for progress bar (thus the download can be canceled by the user, and if the internet connection is “broken” etc, then I can time out gracefully….

I also use the windows api’s to wininet since they are easy to use, don’t add any references (or bulk) to my code, and they work rather well. And, I don’t have to use any activex controls either.

However, I would like to consider using HTTP in place of FTP for my downloads (software updates). However, two problems come up:

1) – I don’t know how to get the file size when using HTTP, and this brings me to #2
2) – without file size, then I can’t make a progress bar..

Further, the ‘worst’ problem is that for ftp downloads, I have to supply (imbed) in my software a logon and password. I really don’t like having that logon and password in my software. 

I assume using HTTP means I can dispense with those logon and passwords (as long as the file I need is exposed via a http url…and, in my case the file IS exposed that way).

I been using wininet and a nice “callback” since I can then cancel the download, and the download is asynchronous (at least in terms of a freeze up…I can get out easy – the software is not threaded, ..but, a call back does mean things don’t hang if it stops). 

So, I guess the question is can I get file sizes for downloads when using HTTP, and further, can I use callbacks with HTTP downloads?

I suppose any other neat-o way to provide updates in my software is also welcome here. (what do you folks use to udpate your software?)

Anyone have a reference on using HTTP for downloads with a call back progress bar?

Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Friday, January 12, 2007
 
 
Speaking from a strictly mechanical point of view, HTTP servers may send a Content-Length header prior to sending the entity body for any resource, so you can base your progress bar on the value of this header.  However, be aware that this header is OPTIONAL ("should" and not "must" in RFC 2616, which you should really read), so you must write code to handle cases where the total size is unknown.

Firefox, for instance, displays the bytes downloaded so far for files with no Content-Length header and a full progress bar with both downloaded and an estimated remaining time for files with the header.

Apache, by default, sends Content-Length headers automatically for static content (files from the filesystem); scripts or server-side applications can also add a Content-Length header to the response themselves.
Angstroem
Friday, January 12, 2007
 
 
Can you not use anonymous FTP? This strikes me as a potential solution to the problem, since you already have FTP support working.
Tom_
Friday, January 12, 2007
 
 
Nice thing about HTTP is it will function in environments like my day job where we can make proxied HTTP to the Internet, but no other request types (including ftp) make it outside.
Doug
Friday, January 12, 2007
 
 
>Can you not use anonymous FTP?

Unfortunately, that is what I am trying to avoid. Anonymous ftp would force me to open up some directories that I don’t want to.

Right now, FTP via password occurs (end uses don’t know, or care where this resolves to). However, I also create a http://URL address for that same file, and this again exposes the file to the web server, which in turn exposes the file to a URL download. However, this does NOT open up or expose the files and directories to everyone on the web.

So, for reasons of security, I don’t allow anonymous ftp right now. In fact, this is much why I am asking for HTTP…as that allows me to expose files, and downloads without the need for a user + password (and, of course also means I don’t have to grant anonymous FTP access to those directories where the file are either). So, only if I create a url link is the file avalable (if I in fact get http working).

I suppose I could consider creating a specific ftp logon id for software only updates, but I still exposing a username + logon for software updates…and I don’t want to do that…


Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Friday, January 12, 2007
 
 
You could always add another document, also available via HTTP, that contained a list of downloadable updates along with file sizes.  Your update tool could download that file first and parse it for the file size.

In general, though, using the content-length header is the better solution.
woot
Friday, January 12, 2007
 
 
>a list of downloadable updates along with file sizes.  Your update tool could download that file first and parse it for the file size.

That is absolutely brilliant. I don’t know if I do that, but, that certainly is a good idea……

Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Friday, January 12, 2007
 
 
If by "good" you mean "terrible".  You're duplicating work HTTP does for you, which means you might do it wrong.  It also means you have to maintain the list, which is one more thing that can go wrong.

From digging around on MSDN, I found
http://msdn2.microsoft.com/en-us/library/aa384238.aspx
which should let you query for the content length header.
Angstroem
Friday, January 12, 2007
 
 
According to:

Best Practices for User Account Control in Windows Vista

"Self Updating Code – DON’T DO IT

"This is our LARGEST App Compat problem Home consumer user applications"

http://devreadiness.org/files/5/presentations/entry5.aspx
Codger
Friday, January 12, 2007
 
 
>From digging around on MSDN, I found

Thanks, that helps a lot, and that gives me the file length (however, the server is on a Linux and is apahce. That command suggests MS’s IIS server is needed. But, I will give it a try…

By the way, the suggestion about the text file idea to get the file length is not something I would likely use. However, the solution is one of those out of the box answers that impressed me!!.

I am kind of torn right now, since the wininet api’s for FTP is really nice (you just pass it a remote file, and local file…and it does the transfer. Throw in a callback setup..and you have a nice progress bar etc.

With HTTP, I have to use InternetReadFile…which means I actually having to write code to take bytes in a buffer, .and write them out to a windows file (that is a bit too low of a level of coding at this point for my tastes). Unless I missed something in the HTTP commands, then it turns out I was dead wrong in thinking that HTTP would be simpler then FTP.

While FTP has issues of logon, passwords, text/binary mode, and is particular about the directory you are in, at least the FTP commands in wininet allows the code to do a file transfer for you.  HTTP has no such file transfer ability, and thus you have receive data into a buffer, and then write out (via a loop) the data as it is received in chunks out to a file in code.

>"Self Updating Code – DON’T DO IT

It is not self updating code. It is full new install that overwrites the old one. Furhter, the user initiates  the download (upgrade). The code does not yet phone home, or do it do a upgrade without the user starting the process.....


Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Friday, January 12, 2007
 
 
I'm confused about why anonymous FTP is bad while HTTP is OK.  Security-wise, it's pretty much the same thing (ignoring port issues).  I understand that you're concerned about opening a directory that contains this update to the world, but if I can pull it via HTTP, then that's anonymous access.  If you grant read-only access to anonymous FTP, that's pretty much the exact same thing as allowing HTTP access.

Or am I missing something?
Dave Send private email
Friday, January 12, 2007
 
 
HTTP is the way to go.  FTP is not a viable solution if you want this to work reliably.  FTP is very commonly blocked in corporate environments.  It's also easy for it to get blocked by software or hardware firewalls.  It's likely that your users use, and thus test, their HTTP connections on a daily basis.  It's just as likely that the vast majority never use FTP.  There's absolutely no advantage to using FTP -- only the huge disadvantage of it being much more likely to not work. 

The text file idea is standard practice.  You'll see this if you sniff for HTTP traffic on your PC with some apps that do auto-updates.  It's common to see something like this:

1.0.5.1
http://example.com/app_1_0_5_1.zip
- Fixes incompatibility with some video drivers.

This gives you the version number so you can determine if an updated version is available (how the heck do you determine this now?).  It also gives the URL of the download which can be convenient for giving you some flexibility in terms of how you name the update and where you put it.  And it lets you put in metadata like release notes or a severity indicator so you can give the user some options as to if they care to update sooner or later. 

Finally, it should be safe to rely on content-length for size.  I don't think I've ever seen the case of a server not including content-length when it's easily predetermined such as with a file transfer.  It should be easy enough to test this with the server you're currently using.
SomeBody Send private email
Saturday, January 13, 2007
 
 
Changing tack slightly but why do you need a progress bar at all. It sounds like you are forcing the user to stop their work while the download takes place. Surely this is best done in a thread or a separate process and you simply notify the user when it is finished.

And yes http is the way to go. You will have to cater for folks in a corporate environment who use proxies though. ie. Provide somewhere to enter the proxy info and then prompt them for a user id and pwd when required.
Neville Franks Send private email
Saturday, January 13, 2007
 
 
HTTP content-length can't be absolutely relied on in all environments, but if your application is downloading a file from your server, then it will be there unless you go out of your way to make it otherwise.
G Jones
Saturday, January 13, 2007
 
 
>but if I can pull it via HTTP, then that's anonymous access

Not quite. In the case of ftp, you are exposing that directory of files to the net. In the case of http, you only exposing that ONE file via a url. You create a URL for the web server to dish it out. So, you still never grant ftp access to users and never expose those files to the web (no ftp = keep people away). So, using HTTP means ftp users can’t see, or use any of the files in that directory. Using HTTP does not open up that directory, and users can’t browse those files. They HAVE to use a SINGLE url to get to that file.

The ONLY file the user can use and download is one that is exposed via a URL. If I give FTP anonymous, then that gives access to the whole directory where those files are. I have things in that client directory that I don’t want to expose. Even some of their custom support web pages (on a client by client bases) are in their ftp directory. I don’t want them to be able to mess with those. I have some files, and things I want that client to be able to download (and, I simply put those in a web page for their support people to install and maintain my software). After the software is installed, I use those same URL’s for updating (well, I will once I get the http update working !!). So, I have other things in that directory that I don’t want the client to see, or have access to. HTTP solves this problem nicely.

>Changing tack slightly but why do you need a progress bar at all.

If I had different tools, and different situation, then sure, no progress bar would be needed. However, the software update is initiated by the user, and, after the download, they do get the familiar “install” dialogs. (I using inno installer).  The software update is OFTEN done while they are talking to support, so, we do want a visual feedback as to how this update is going along.

Further, I don’t have a threaded environment to run this on. Most importantly, that progress bar is fed by setting up a windows api for the wininet library. The callback that allows the progress bar also makes this a asynchronous transfer. So, with callbacks, the transfer is asynchronous.  If you don’t use callback, then if the download hangs…so does your code. Further, that update to the software only takes about 25 seconds.

So, the progress bar provides a nice way to tell support people how things are going, and also to the user that might be installing this update as per instructions via a email.

When I “often” test this download and update code, I pull the network plug halfway in the middle of the download.  It times out very nicely, and the code simply moves on to giving the user a nice message that the download could NOT complete.

To be fair, the  “callback” api code could be used, but not necessary feed the progress bar. However, I might as well have it do so since the user has to wait and watch the process anyway.

If you don’t use a call back, then in effect you forced to use threaded code (which I don’t have), or launch a separate application to do the update (which is another piece of software to build, install, run and maintain).

In either of the above cases, if the download hangs, you still have a process, or piece code that has be killed, or stopped. So, I keep it simple and have the user as part of this process. If the connection fails during download…it fails gracefully right in front of the user, and the user does not even exit the software application. A dialog box simply tells them the update could not be done, they click ok..and are still in the application…..and can go back to work. I like this.

My code knows the download has completed, and then I shell out to the new .exe install, and do a application quit. That application that was *JUST* running is about to be over written.

Reducing the number of processes, parts to install, and to maintain keeps this simple and makes it reliable.

If I use a separate update program, then that program has to ensure that the main program is not running before the install starts. And, if the main program is running, then do I kill that main program? (can’t do that..might be running a large printout, or important year end process). So, now, do I ask the user what to do? The spec sheet as to what to do to kill a simple program running on windows runs MANY PAGES long!! What kind of dialogs do we even give the user? How do I know if the main program is running by accident, or on purpose? This complex list of issues goes on and on. I could ask the user to exit the main program...but, with my setup..I don't have to!!

So, just even what kinds of dialogs and prompts you give to the user to deal with this is at issue. A simple update now becomes VERY complex as to what you do. So, introducing some update system that runs independent of the actual software is a rather daunting task.

So, a separate program (or thread) that does the update could allow me to get rid of the progress bar, but then that separate program (thatt cost developer $$) has to figure out how to QUIT the main running program and deal with several issues is actually quite complex.

Right now, keeping the update code in the MAIN program keeps this whole process is quite simple. It is rather minimal in terms of development resources and costs. As mentioned, I really don’t have the software setup to run the download in background anyway. So, I kind of stuck with a progress bar with the current environment. However, even if I could do this as a background download, I don’t think I would..as that type of setup brings in a enormous amount of a additional issues as mentioned.

So, that is why I use a progress bar.

If I have to spend anymore time then the simple ftp and callback I have developed now, then I am likely consider purchasing something.

I just want to change my ftp code over to http, and be done with this.


Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Saturday, January 13, 2007
 
 
"If I give FTP anonymous, then that gives access to the whole directory where those files are. I have things in that client directory that I don’t want to expose. Even some of their custom support web pages (on a client by client bases) are in their ftp directory."

You seem to think that your anonymous FTP account has to point to the same directory as your regular account -- it doesn't.  Just create a new empty directory called "update", set anonymous access to be root at that directory and put your file in there.
Flame on Send private email
Saturday, January 13, 2007
 
 
> It is not self updating code. It is full
> new install that overwrites the old one.
> Furhter, the user initiates  the
> download (upgrade). The code does not
> yet phone home, or do it do a upgrade
> without the user starting the process.....

I think you missed the point.

"Don't have a "Self-Update" that is wanting to put things in program files or other areas that a standard user doesn't have rights too.  This won't work and unfortunatly lots of software does this these days."

http://windowsconnected.com/blogs/joshs_blog/archive/2006/03/30/1519.aspx


The story we're hearing on Vista (I don't have a copy running to test on) is that users can't write to Program Files etc. anymore.  Not even Admin account users.

Some ISVs seem to be mucking things up more by installing into Common Files instead, which is meant for box-level config data and not code.  Can't wait to see how things play once MS makes it impossible to execute from there in SP1! [sarcasm, not announced that I have heard of]
Codger
Sunday, January 14, 2007
 
 
You might also watch (or read the transcript to) The .NET Show #59 Windows Vista Readiness.

http://msdn.microsoft.com/theshow/archive.aspx
Justinian
Sunday, January 14, 2007
 
 
>The story we're hearing on Vista (I don't have a copy running to test on) is that users can't write to Program Files etc. anymore.  Not even Admin account users.

Ah...Yes I see you point!!

I am well aware of this. In fact, I seeing some clients now lock up the "program files" area in xp also. (only read access).

This started happening a new of years ago, and so I much so, that I am ready for this change.

For example, my update and program stuff used to be in

C:\program files\myAppDir\

Now,  moved this to:

C:\Documents and Settings\All Users\Application Data\MyAppDir\

Interesting, I actually asked some support people about what directory they would prefer I use in the above case.

Amazing…but a good number of support people are either blank faced when I ask them, or simply state that program files is a good place. I finding out that program files is much off limits for any program data now….and has been for some time….

As a few mentioned, when the above directory does not allow .exe files, or the above dir goes read only…I am in big trouble!!!!

Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Monday, January 15, 2007
 
 
> As a few mentioned, when the above directory does not allow .exe files,
> or the above dir goes read only…I am in big trouble!!!!

You won't be alone!

Still trying to get a good handle on the "right" way to self-update in Vista here.  Your environment seems to be VBA.  I assume you'll have similar challenges to those of VB programmers.

Right now I'm looking at priviledge elevation through an out of process COM library, probably an ActiveX EXE.  Until I have a Vista testbed though I can't be sure I have a working solution.
Tasked Without Budget
Monday, January 15, 2007
 
 
I have a much less elegant but working solution.

When the user asks my programs to check for updates the program checks if there are a new version. If there are one it ask the users if he want to visit a webpage where he can read more about it and download the new version.
Martin Schultz Send private email
Thursday, January 18, 2007
 
 
"The story we're hearing on Vista (I don't have a copy running to test on) is that users can't write to Program Files etc. anymore.  Not even Admin account users."

This is not correct.  I have a couple of programs that insist on writing to their home directory (C:\Program Files\foo\bar).  To make then run in Vista, I just give myself full Admin read/write permission for those directories.  It can be done.
Dick Peter Johnson
Friday, January 26, 2007
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz