Inexplicable ASP.NET Slowdowns

Inexplicable ASP.NET Slowdowns
   I believe every developer has been in a situation where the software behaves in a way that just cannot be.  Every developer has applied fixes that do not make sense and he had no idea how they worked. Recently I came across the most brutal version of "cannot be" I had ever encountered while working on software.

   In the project I work on users can attach images and other files in comments to various items. When the user uploads a file a preview of the image is displayed and then the user can post the comment. This is all done via AJAX. When the comment is posted the server returns HTML for the comment which is displayed on the page. When the user uploads a file a temporary folder is created for the comment and the images and their thumbnails are stored in it. The preview images are loaded from this folder and when the comment is posted it is stored in the database together with the images. The temporary folder is then deleted and the HTML for the comment is returned to the client. The images for the actual comment are loaded from the database. As you may know I am a big advocate of storing user files in the database and this includes temporary files but this was not my decision. In fact we moved the files to the database after the whole upload was already written. The original version copied the temporary files to a new location on the file system.

   The first problem we ran into was that sometimes the delete of the folder failed probably due to some OS file lock. My solution was to catch the IO exception sleep the thread for 10 milliseconds and try again. Sometimes very rarely the delete in the catch would still throw an exception. This is why I added a second try/catch inside the catch block and forced the thread into a sleep for a whole second. It seems like the file lock issue is quite common. Even Google Chrome has similar solution to the problem (it tries to delete the file twice).

   However we noticed something even stranger. When a comment was posted the text appeared immediately but there was a significant lag before the image was displayed. This only happens the first time a comment is displayed. Of course my code was blamed because it forced a thread into a sleep. I was quick to point out that my code (including the sleep) was already executed when the HTML is sent back to the client and the text of the comment did appear immediately. It was the image request that was lagging. Clearly my code could not cause that. A coworker of mine commented out the whole code responsible for deleting the temporary folders and… the images were loading fast… WTF?!?!? He was quick to refactor the code to use the wonderful Task Parallel Library to delay the deleting of the temporary folder by scheduling a task on another thread. As cool as TPL is I stopped him and demanded that we found out why this seemingly impossible thing was happening instead of hacking around it. I just did not believe that a sleep in one request can cause the next request to lag. What is more the second request was only triggered when the first request had returned because the link to the image is in the HTML that the first request returns.

   After several hours of commenting out random lines of code, changing sleep timings and blank staring at the code I noticed that after the AJAX request was finished and before the request for the image began the Visual Studio status bar went crazy looping through different dlls and then I had the eureka moment. Because the folder structure of the project had changed (i.e. a folder was deleted) IIS had to recompile the project. The recompilation triggered by the delete and not the sleep caused the delay. Now that I think of it I am so glad I insisted on investigating the cause of the issue. If we had implemented a delay for deleting the temporary folder it would appear that the problem was solved because the image would appear instantly. However after the delay the folder would be deleted and we would still get a delay. This delay would have been much worse because we would not be able to link it to attaching files. It would have been seemingly random inexplicable delay for the whole system.

   As much as I would like I cannot add this issue to the list of arguments for storing user files in the database instead of on the file system. My quick search did not find easy way to disable recompilation triggered by certain folder but even if it did I would still not use it. Even when you store user files on the file system you should pull the folder out of the website folder structure. This simplifies updates to the code and backup of the user files. Of course you should never store user files on the file system but even more importantly you should always know the answer to the question "Why?".
Tags:   english programming 
Posted by:   Stilgar
13:24 28.06.2011

Comments:

First Previous 1 Next Last 

Posted by   JOKe (Unregistered)   on   14:20 28.06.2011

I can only say that this deleting files from the FS is really a crap.. as you said Chrome also have it.. but the issue appears only on crappy OSes like Windows.. In the real world no one is using .net and windows for something big so no issue :) (I was having the same issue on my dev machine but no issues on normal os) .
Still I have a question .. WHO SICK MIND have come to the idea to put this temp folder inside the project ? FIRE HIM !!! I DEMAND !

P.S. also I have a question because.. I find that this temp folders are very COMMON in the .net world ... I've used to "fix" a .net project where someone will not say who was creating a temp folder like in your case (not in the project folder.. fiu   but still the folder name was some sort of DateTime.Now.Millisecond + DateTime.Now.Second + etc.. you get the idea... WTF ? :D it is totally normal to have more then one request at the same ms .. so this was causing a big issues on production .... ( more then 3000 users around 6 o'clock every day and some of them doing "this" request at the same ms)

Posted by   Stilgar   on   15:23 28.06.2011

I guess you never ever use Stack Overflow :)

I believe the folder ends up in the project structure because it is easier to develop this way and you don't need to setup virtual directory in IIS and grant permissions as you need to do when the folder is outside the project structure. Basically people do it because they are lazy. Hell, even I am lazy and the large images in the articles are stored in the project structure. To my defense BlogNET does not have functionality for file uploads yet (except for the images in the left corner in the articles) and I am using FTP to upload other files so technically these are not user files.

I don't want to comment further on the best way to store user files on the file system because this would imply that there is a right way to store user files on the file system and there is not.

Posted by   JOKe (Unregistered)   on   15:40 28.06.2011

you can always store files in /tmp :D ops no sorry you cant ahh windows sorry about that ..

Posted by   Stilgar   on   15:54 28.06.2011

I would fully approve if on Windows you could not write to the file system from a web application because it is always wrong. BTW if you want to write to /tmp you could (and should) use Isolated Storage ( http://msdn.microsoft.com/en-us/library/3ak841sy.aspx ) and guess what it would work the same on Windows, Mac and Linux via Mono. Of course there is no reason for a web app to use Isolated Storage... after all it already has an isolated storage location. It is called the database.

Posted by   linker (Unregistered)   on   16:36 28.06.2011

Even if you have no other choice but use a temp folder, why would you use different temporary folders every time? Btw, when deleting files in windows explorer it tries to delete the file several times before giving an error message.

Posted by   Stilgar   on   16:49 28.06.2011

@linker there can be multiple files for each comment and evem more there are different groups of files. For example you can attach a video and a thumbnail is extracted from the video. If you don't know about the folder issue it actually makes sense to organize stuff in folders instead of encoding all the info in the file name. Of course storing user files on the FS does not make sense in the first place.

First Previous 1 Next Last 


Post as:



Post a comment: