Writing Server-Side Upload Code

Upload Processing Approaches

Reading the Entire Request at Once

The most obvious way to provide access to uploaded data from the upload processing page is to receive the entire request and after that fill a special memory structure with elements of the request. Usually upload processing facilities provide two collections: Form and Files. The first one contains a collection of simple POST fields (integer and string values), the second one contains files.

The advantage of such an approach is that after you fill these collections, you have an access to any element of the request at any moment. You do not care in what order the files and variables were sent. In a number of cases it makes script logic clear and coding easier.

However, the price for it is a memory overhead. It becomes more noticeable when you upload a lot of files and these files are big. Usually you do not need to have all files at one time because you process files one-by-one (save them to disk or carry out some else operations). But you have to load all files into memory, and it makes your server to work much slower, especially if multiple users are uploading files simultaneously.

You can eliminate this problem by uploading files in separate requests as discussed in the Uploading Files Sequentially and Concurrently topic. It will reduce the problem if multiple small files are uploaded; however, it will not be helpful enough if few huge files are uploaded (e.g. video movies, etc).

In some implementations of the server-side upload facilities this problem is solved by storing uploaded files on a disk instead of memory. Classic example of this approach is PHP. Also, ASP.NET 2.0 allows configuring memory buffer threshold. When it is exceeded, the request is flushed to the file.

Reading the Request Incrementally

A more complex, but more efficient upload processing technique is to process the data while receiving the request (in other words, incrementally). The crucial difference with the previous approach is that you work with the uploaded request as with a stream rather than a collection of elements.

It means that you cannot rewind to the files which were processed before and go forward to files which are not yet processed. But you do not have to wait until all files are received, and more importantly, you do not have memory overhead.

So incremental upload processing is more difficult to use, but the performance is noticeably better. That is why if you expect that users will upload a large amount of data, we strongly recommend you to consider this technique in your upload processing script.

This stream-based uploading technique is implemented with some ASP components, for example - Dundas Upload component. All ASP demo applications use Dundas component in this way.

Another platform where it can be used is ASP.NET, but it will require that you use your own HttpModule instead of standard uploading classes.

Request Integrity Checking

Even through the user aborts uploading, the upload script is run and receives an incomplete request. Such requests may cause unexpected errors in your upload script. For example, the script may try to save a thumbnail which is not actually uploaded due to cancellation. To avoid this problem, it is recommended to add request integrity check in your upload script before parsing the request and saving uploaded files.

To perform this check you can use the FileCount POST field which is sent by Image Uploader at the end of the upload request. Thus, if this field is not received the upload is considered as cancelled.

The code samples below demonstrate how this check can be done in ASP.NET and PHP.

private void Page_Load(System.Object sender, System.EventArgs e)
{
	// Check that all files are uploaded
	if (HttpContext.Current.Request.Form["FileCount"] != null)
	{
		// ...
		// Iterate through uploaded data and save the original file and thumbnail
		// ...
	}
}

// Check that all files are uploaded
if (array_key_exists('FileCount', $_POST))
{
	// ...
	// Iterate through uploaded data and save the original file and thumbnail
	// ...
}

The alternative way is to use Image Uploader ASP.NET control or PHP Library which both perform the request integrity checking internally.

Uploading in ASP

ASP does not have any standard file upload processing facilities, and that is why there are a lot of third-party ActiveX components for this purpose available. Most components implement the simple uploading technique of loading entire request into memory; however, there are components which can process the request incrementally.

The following code sample demonstrates incremental uploading technique of the Dundas Upload component.

When you call the GetNextFile method, the component uploads data until the first uploaded file is encountered. This way it fills the Form collection with variables, which are sent before the file, but the file itself is not requested. As soon as you call the Save method, the file is actually uploaded and saved directly to disk (without storing it in memory). Then the GetNextFile and Save calls are repeated for all other files.

<%@ Language="VBScript" %>
<%
'This variable specifies relative path to the folder, where the gallery 
'with uploaded files is located.
Dim strGalleryPath
strGalleryPath = "../Gallery/"

'Create Dundas Upload object to process uploaded images.
Dim objUpload
Set objUpload = Server.CreateObject("Dundas.Upload.2")
objUpload.UseVirtualDir = False
objUpload.MaxUploadSize = 1000000000
'First call of GetNextFiles will read the upload stream until the first file
'and fill the Form collection (in particular you will get FileCount variable).
Dim objNextFile
Set objNextFile = objUpload.GetNextFile

'Get total number of uploaded files (all files are uploaded in a single package).
Dim intFileCount, i
intFileCount = objUpload.Form("FileCount").Value

'Iterate through uploaded data and save the original file, thumbnail, and description.
For i=1 To intFileCount
	'Get source file and save it to disk.
    If i > 1 Then 
        Set objNextFile = objUpload.GetNextFile
    End If
    objNextFile.Save Server.MapPath(strGalleryPath)  
    'Get original file name.
	Dim strFileName    
    strFileName = objUpload.GetFileName(objNextFile.OriginalPath)
    'Rename saved file to original name.
    objUpload.FileMove Server.MapPath(strGalleryPath & objNextFile.FileName), _
        Server.MapPath(strGalleryPath & strFileName)     
	'Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
    Set objNextFile = objUpload.GetNextFile
    objNextFile.Save Server.MapPath(strGalleryPath & "Thumbnails/")
    'Rename it to original name.
    objUpload.FileMove Server.MapPath(strGalleryPath & "Thumbnails/" & objNextFile.FileName), _
        Server.MapPath(strGalleryPath & "Thumbnails/" & strFileName & ".jpg")     
Next
%>

Uploading in ASP.NET

To process uploaded data in ASP.NET, you can use intrinsic object named Request (System.Web.HttpRequest class).

Before Page_Load method is called, ASP.NET initializes the Request object (i.e. uploads entire POST request and fills the Form and Files collection).

ASP.NET versions earlier than 2.0 stored the request in memory. This way it was quite inefficient with huge uploads. But in ASP.NET 2.0 you can set the threshold value for the upload buffering. If the POST request length exceeds this value, the request is flushed to the temporary file.

This threshold is specified in the web.config file of your application with the requestLengthDiskThreshold attribute of the httpRuntime element. The threshold is specified in bytes. See the httpRuntime Element (ASP.NET Settings Schema) MSDN article for more information about it.

Also, you can create a custom HttpModule, which saves uploaded data to disk, or implements the incremental upload technique.

<%@ Page Language="VB" AutoEventWireup="false" ValidateRequest="false" %>
<script runat="server">
'This variable specifies relative path to the folder, where the gallery 
'with uploaded files is located.
Private galleryPath As String = "../Gallery/"

Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
	'Get total number of uploaded files (all files are uploaded in a single package).
	Dim fileCount As integer = Int32.Parse(Request.Form("FileCount"))

	'Iterate through uploaded data and save the original file and thumbnail
	For I As Integer = 1 To fileCount
		'Get source file and save it to disk.
		Dim sourceFile As HttpPostedFile = Request.Files("SourceFile_" & I)
		Dim fileName As String = System.IO.Path.GetFileName(sourceFile.FileName)
		sourceFile.SaveAs(Server.MapPath(galleryPath & fileName))

		'Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
		Dim thumbnail1File As HttpPostedFile = Request.Files("Thumbnail1_" & I)
		thumbnail1File.SaveAs(Server.MapPath(galleryPath & "Thumbnails/" & fileName & ".jpg"))
	Next
End Sub
</script>

<%@ Page Language="C#" AutoEventWireup="true" ValidateRequest="false" %>
<script runat="server">
//This variable specifies relative path to the folder, where the gallery 
//with uploaded files is located.
private string galleryPath = "../Gallery/";

private void Page_Load(System.Object sender, System.EventArgs e)
{
	//Get total number of uploaded files (all files are uploaded in a single package).
	int fileCount = Int32.Parse(Request.Form["FileCount"]);

	//Iterate through uploaded data and save the original file and thumbnail
	for (int i = 1; i <= fileCount; i++)
	{
		//Get source file and save it to disk.
		HttpPostedFile sourceFile = Request.Files["SourceFile_" + i];
		string fileName = System.IO.Path.GetFileName(sourceFile.FileName);
		sourceFile.SaveAs(Server.MapPath(galleryPath + fileName));

		//Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
		HttpPostedFile thumbnail1File = Request.Files["Thumbnail1_" + i];
		thumbnail1File.SaveAs(Server.MapPath(galleryPath + "Thumbnails/" + fileName + ".jpg"));
	}
}
</script>

Uploading in ColdFusion

ColdFusion exposes easy-to-use upload handling. The tag named <cffile> provides all necessary functionality. In its attributes you specify the field name, the folder where you are going to save it, and what to do in case of file name collision.

To get access to form variables, use the following syntax:

#FORM.<field name>#

For example, to get the file count, use:

#FORM.FILECOUNT#

An example of ColdFusion script that processes the upload is presented below.

<cfprocessingdirective pageEncoding="utf-8">

<!---This variable specifies relative path to the folder, where the gallery
with uploaded files is located.--->
<cfset galleryPath="../Gallery/" />
		
<cfset absGalleryPath="#ExpandPath(galleryPath)#" />
<cfset absThumbnailsPath="#absGalleryPath#Thumbnails/" />
			

<!---Get total number of uploaded files (all files are uploaded in a single package) and
iterate through uploaded data and save the original file, thumbnail, and description.--->
<cfloop index="i" from="1" to="#Form.FileCount#">

	<!--- Get source file and save it to disk. --->
	<cffile action="UPLOAD" filefield="SourceFile_#i#" 
		destination="#absGalleryPath#" 
		nameconflict="MakeUnique">
				
	<cfset fileName="#serverFile#">

	<!--- Get first thumbnail (the single thumbnail in this code sample) and save it to disk. --->
	<cffile action="UPLOAD" filefield="Thumbnail1_#i#" 
		destination="#absThumbnailsPath#" 
		nameconflict="MakeUnique">

	<!--- Rename thumbnail file so that it has .jpg extension --->
	<cffile action="rename"
		source="#absThumbnailsPath#/#serverFile#"
		destination="#absThumbnailsPath#/#fileName#.jpg"> 

</cfloop>

Uploading in JSP

Different JSP servers may provide different means to process uploads. Let's examine how to do it on a popular JSP server Tomcat.

There is no built-in upload processing facility for Tomcat. However, there is an open-source package called Jakarta Commons FileUpload developed by the Apache Software Foundation. You can find the most recent version at this location:

http://jakarta.apache.org/commons/fileupload/

It works in the similar way as most ASP components or built-in ASP.NET class. When you call the parseRequest method of the DiskFileUpload class, it returns you a list of FileItem class instances that represent uploaded files and POST fields. You can save files to disk, write additional data to database, or carry out any other operations.

See the code example below.

<%@ page language="java" import="java.io.*,java.util.*,javax.servlet.*,
	javax.servlet.http.*,org.apache.commons.fileupload.*"%>
<%
//This variable specifies relative path to the folder, where the gallery 
//with uploaded files is located.

String galleryPath = "/Gallery/";

//Process upload.
ServletContext context = getServletContext();
String absGalleryPath = context.getRealPath(galleryPath);
String absThumbnailsPath = context.getRealPath(galleryPath + "/Thumbnails");
String absTempPath = context.getRealPath(galleryPath + "/Temp");

DiskFileUpload fu = new DiskFileUpload();
//Set maximum size before a FileUploadException will be thrown.
fu.setSizeMax(100000000);
//Set maximum size that will be stored in memory.
fu.setSizeThreshold(4096);
//Set the location for saving data that is larger than getSizeThreshold().
fu.setRepositoryPath(absTempPath);

//Get uploaded files.
List listFileItems = fu.parseRequest(request);

//Put them in hash table for fast access.
Hashtable fileItems = new Hashtable();

for (int i = 0; i < listFileItems.size(); i++) {
    FileItem fileItem = (FileItem)(listFileItems.get(i));
    fileItems.put(fileItem.getFieldName(), fileItem);
}

//Get total number of uploaded files (all files are uploaded in a single package).
int fileCount = Integer.parseInt(((FileItem) fileItems.get("FileCount")).getString());

//Iterate through uploaded data and save the original file, thumbnail, and description.
for (int i = 1; i <= fileCount; i++) {
    //Get source file and save it to disk.
    FileItem sourceFileItem = (FileItem) fileItems.get("SourceFile_" + Integer.toString(i));
    String fileName = new File(sourceFileItem.getName()).getName();
    File sourceFile = new File(absGalleryPath + File.separator + fileName);
    sourceFileItem.write(sourceFile);
    
    //Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
    FileItem thumbnail1FileItem = (FileItem) fileItems.get("Thumbnail1_" + Integer.toString(i));
    File thumbnail1File = new File(absThumbnailsPath + File.separator + fileName + ".jpg");
    thumbnail1FileItem.write(thumbnail1File);
}
%>

Uploading in PHP

PHP comprises convenient upload processing facility. There are two built-in collections:

• $_POST (used to be called $HTTP_POST_VARS in earlier versions of PHP) - contains form variables.
• $_FILES (used to be called $HTTP_POST_FILES in earlier versions of PHP) - contains files.

These collections are filled before the PHP script is started. The good news is that files are stored in temporary files on disk instead of memory, and, therefore, your server will not suffer from memory shortage during large uploads.

<?php
//This variable specifies relative path to the folder, where the gallery 
//with uploaded files is located.
$galleryPath = "../Gallery/";

$absGalleryPath = realpath($galleryPath) . "/";

//Get total number of uploaded files (all files are uploaded in a single package).
$fileCount = $_POST ["FileCount"];

//Iterate through uploaded data and save the original file and thumbnail.
for ($i = 1; $i <= $fileCount; $i++)
{
	//Get source file and save it to disk.
	$sourceFileField = "SourceFile_" . $i;
	$fileName = $_FILES[$sourceFileField]["name"];
	move_uploaded_file($_FILES[$sourceFileField]["tmp_name"], 
		$absGalleryPath . "/" . $fileName);

	//Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
	$thumbnail1Field = "Thumbnail1_" . $i;
	move_uploaded_file($_FILES[$thumbnail1Field]["tmp_name"], 
		$absGalleryPath . "/Thumbnails/" . $fileName . ".jpg");
}
?>

Uploading in Perl

In general, when you work in Perl you should manually parse the POST request. When the server receives the request, it writes all uploaded data into an input stream.

However, fortunately, you can use additional modules where this functionality is already implemented. In particular you can use standard CGI module, which provides access to multipart/form-data fields using the param() routine. To obtain form variables values, just pass the appropriate field name to this function (for example $FileCount = param('FileCount')).

To receive a file, you should undertake some additional actions. When you call param('SomeFileFieldName'), you get not only file content, but also additional data appended to the beginning of the file: content-disposition, filename, and content-type. You should extract the filename (using the basename function) and skip unnecessary data to start copying the file content to disk.

The code sample below demonstrates this.

#!/usr/bin/perl

use CGI qw/:standard/;
use File::Spec;
use File::Basename;

#This variable specifies relative path to the folder, where the gallery 
#with uploaded files is located.
my $galleryPath = "../Gallery/";

#Process upload.

print "Content-Type: text/html\r\n\r\n\r\n";

my $absGalleryPath = File::Spec->rel2abs($galleryPath);
my $absThumbnailsPath = File::Spec->join($absGalleryPath, "Thumbnails");
	
#Get total number of uploaded files (all files are uploaded in a single package).
my $fileCount = param('FileCount');
	
#Iterate through uploaded data and save the original file, thumbnail, and description.
for (my $i = 1; $i <= $fileCount; $i++){
	#Get source file and save it to disk.
	my $sourceFilePath = param("SourceFile_" . $i);
	my $fileName = basename($sourceFilePath);

        #Copy file to gallery folder
	open(OUT, ">" . File::Spec->join($absGalleryPath, $fileName)); 
	binmode(OUT);  
	while (<$sourceFilePath>) 
	{   
		print OUT $_;   
	}  
		close(OUT);	
               	       
	#Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
	my $thumbnail1Path = param("Thumbnail1_" . $i);
	#Copy file to thumbnails folder
	open(OUT, ">" . File::Spec->join($absThumbnailsPath, $fileName . ".jpg")); 
	binmode(OUT);  
	while (<$thumbnail1Path>) 
	{   
		print OUT $_;   
	}  
	close(OUT);      	
}

Uploading in Python

To work with HTTP requests, Python includes a cgi module. This module exposes the FieldStorage class that represents a list of all uploaded POST fields (both files and form variables). This list is filled before the page is run automatically.

There is one interesting feature in Python. It accepts several POST fields with the same name (unlike some server platforms that require each field to have a unique name). Such fields are returned as a nested list. Although Image Uploader does not send fields with the same name, you should keep in mind this behavior.

This code example demonstrates how to process the upload in Python.

#!/usr/local/bin/python

import cgi
import cgitb; cgitb.enable()
import os

try: # Windows needs stdio set for binary mode.
	import msvcrt
	msvcrt.setmode ( 0, os.O_BINARY ) # stdin  = 0
	msvcrt.setmode ( 1, os.O_BINARY ) # stdout = 1
except ImportError:
	pass

#This variable specifies relative path to the folder, where the gallery 
#with uploaded files is located.
galleryPath = "../Gallery/"

#Process upload.

print "Content-Type: text/html\r\n\r\n\r\n"

absGalleryPath = os.path.abspath( galleryPath )
absThumbnailsPath = os.path.join( absGalleryPath, "Thumbnails" )

form = cgi.FieldStorage()

#Get total number of uploaded files (all files are uploaded in a single package).
fileCount = int( form.getfirst( "FileCount", "0" ) );

#Iterate through uploaded data and save the original file and thumbnail.
for i in range( 1, fileCount + 1 ):
	#Get source file and save it to disk.
	sourceFile = form["SourceFile_" + str( i )];			
	fileName = os.path.basename( sourceFile.filename )
	f = open ( os.path.join( absGalleryPath, fileName ), 'wb' )
	while 1:
		chunk = sourceFile.file.read( 10000 )
		if not chunk: 
			break	
		f.write( chunk )
	f.close()
		
	#Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
	thumbnail1File = form["Thumbnail1_" + str( i )];
	f = open ( os.path.join( absThumbnailsPath, fileName + ".jpg" ), 'wb' )
	while 1:
		chunk = thumbnail1File.file.read( 10000 )
		if not chunk: 
			break	
		f.write( chunk )
	f.close()

Uploading in Ruby

Upload processing in Ruby is similar to Python. It also has a CGI library that is used to work with HTTP requests.

First of all, you need to get an instance of cgi object by using the CGI.new() call. The cgi object exposes the params property that presents the collection of POST fields (both files and form variables).

There is one interesting feature in Ruby. It accepts several POST fields with the same name (unlike some server platforms that require each field to have a unique name). Such fields are returned as a nested list. Although Image Uploader does not send fields with the same name, you should keep in mind this behavior.

This code example demonstrates how to process the upload in Ruby.

require 'cgi'

#This variable specifies relative path to the folder, where the gallery 
#with uploaded files is located.
galleryPath = "../Gallery/"

#Process upload.

print "Content-Type: text/html\r\n\r\n\r\n"

absGalleryPath = File.expand_path(galleryPath)
absThumbnailsPath = File.join(absGalleryPath, "Thumbnails")

cgi = CGI.new()

#Get total number of uploaded files (all files are uploaded in a single package).
fileCount = cgi.params["FileCount"].first.read

#Iterate through uploaded data and save the original file, thumbnail, and description.
for i in 1..fileCount.to_i
	#Get source file and save it to disk.
	sourceFile = cgi.params["SourceFile_" + i.to_s].first
	fileName = File.basename(sourceFile.original_filename)
	sourceFileTargetPath = File.join(absGalleryPath, fileName)
	File.open(sourceFileTargetPath.untaint, 'wb') { |file| file << sourceFile.read}

	#Get first thumbnail (the single thumbnail in this code sample) and save it to disk.
	thumbnail1 = cgi.params["Thumbnail1_" + i.to_s].first
	thumbnail1TargetPath = File.join(absThumbnailsPath, fileName)
	File.open(thumbnail1TargetPath.untaint, 'wb') { |file| file << thumbnail1.read}
end

See Also

Reference

• POST Field Reference
• Action Property
• AdditionalFormName Property
• AddField() Method
• RemoveField() Method

Samples

• Samples by Application
• Samples by Platforms

Manual

• Uploading Files Sequentially and Concurrently
• Uploading Additional Data with Files
• Adding Authentication

Troubleshooting

• FAQ: Upload Problems