A utility to help porting Windows .NET applications to Mono/Unix

by grendel 21. December 2009 08:43

Those of you who work or worked on porting Windows applications (any kind of them - ASP.NET, WinForms, console) might have come across a problem with case-sensitivity of file names. Windows file systems are either case-unaware and case-insensitive (the FAT family) or case-aware but case-insensitive (NTFS). In both cases applications performing I/O don't need to worry about using proper file name casing, as the underlying operating system will ignore the case and open the file without problems. This is not the case on Unix operating systems which have file systems that are both case-aware and case-sensitive.

Mono has had a solution for this problem for a long time - it's called IOMAP. It's a runtime mechanism which, if enabled by exporting the MONO_IOMAP variable, will translate not only file name case to the correct form used on disk, but will also strip DOS drive designators from the paths and translate DOS path separator character to the Unix one, thus making the application work seamlessly. The convenience comes at a price, of course, as translating the file names involves performing a lot of disk I/O to discover the real file name. So if your application is supposed to be ran on Unix most (if not all) of the time then the better to dive into the source and make sure that file names on disk are consistent with their use in the source code.

With small applications the operation is usually straight-forward and takes little time. With bigger codebases, however, it might pose a problem, especially if strings used to access files are constructed in different places of the application as opposed to being just plain literals.

For that reason I have recently created a Mono profiler module (it is somewhat of a misnomer, as the utility has little to do with code profiling) which aims at helping the developer/porter to find places in code which call .NET I/O routines passing them misformed file/directory paths as well as identify places where those strings are constructed. The former part is very simple and it merely prints to the console a stack trace leading to the I/O routine call site every time file/directory name mapping is performed. The latter part, however, is a bit more problematic as it has to deal with two separate moments in string's life - its creation and its actual use. The code uses the Mono profiler API to monitor string allocations storing all the strings created in hash tables as well as remembering the stack frame which leads to the string creation site. The code which does that is pretty fast, so it doesn't impact your application's performance too much even though it collects and stores a large amount of data (mostly pointers and some strings, though). When file name mapping is performed during file I/O, the profiler code looks up the string address and retrieves the stack trace to store it for later use. When the application exits a summary report is printed to the console which includes some statistics on the string, its original (requested) and target (mapped) form as well as a location (if it was possible to determine it) where the string was created. The location is determined using simple heuristics, so it might sometimes point to a location which is near the place where the string was created. The heuristics code walks the stack frames looking for first frame which is in application code - that is it ignores the well-known class library assemblies shipped with Mono (corlib, System*, some Mono* assemblies etc) and all the assemblies installed in the GAC. The first frame which doesn't belong in either of the above is considered to be user's code and is reported to be the string allocation location. If the stack trace doesn't contain any such frames, full trace is recorded and shown in the summary.

To take advantage of this utility you will need Mono from trunk (or 2.8 when it is released). It is advisable to compile your application with full debugging information, so that source files and line numbers can be reported (otherwise only Namespace.Class.Method will be printed). To enable the utility, make sure the MONO_IOMAP environment variable is present and set to 'all', 'case' or 'drive' and execute your application as follows:

   mono --debug --profile=iomap your_application.exe

If you're porting an ASP.NET application, use the following command line:

   MONO_OPTIONS="--debug --profile=iomap" xsp2

Below you can see output from a sample application showing the utility's behavior:

 

$ MONO_IOMAP=all mono --profile=iomap --debug iomap-report-sample.exe 
Running test: Mismatched file name case.
-=-=-=-=-=-=- MONO_IOMAP REPORT -=-=-=-=-=-=-
 - Requested file path: 'files/Test2.txt'
 -     Found file path: 'Files/test2.txt'

-= Stack Trace =-
   at System.Environment.get_StackTrace() in /usr/src/tmp/mono/mcs/class/corlib/System/Environment.cs:line 183
   at System.IO.MonoIO.GetFileAttributes(System.String , MonoIOError ByRef )
   at System.IO.MonoIO.ExistsDirectory(System.String path, MonoIOError ByRef error) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/MonoIO.cs:line 252
   at System.IO.Directory.Exists(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/Directory.cs:line 201
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/FileStream.cs:line 246
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 331
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/StreamReader.cs:line 171
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding)
   at System.IO.File.ReadAllText(System.String path, System.Text.Encoding encoding) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 542
   at System.IO.File.ReadAllText(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 537
   at app.MismatchedFileNames() in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 45
   at app.RunTest(System.String banner, System.Action test) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 24
   at app.Main(System.String[] args) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 39

-=-=-=-=-=-=- MONO_IOMAP REPORT -=-=-=-=-=-=-
 - Requested file path: '/home/grendel/Projects/work/iomap-report-sample/files'
 -     Found file path: '/home/grendel/Projects/work/iomap-report-sample/Files'

-= Stack Trace =-
   at System.Environment.get_StackTrace() in /usr/src/tmp/mono/mcs/class/corlib/System/Environment.cs:line 183
   at System.IO.MonoIO.GetFileAttributes(System.String , MonoIOError ByRef )
   at System.IO.MonoIO.ExistsDirectory(System.String path, MonoIOError ByRef error) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/MonoIO.cs:line 252
   at System.IO.Directory.Exists(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/Directory.cs:line 201
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/FileStream.cs:line 270
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 331
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/StreamReader.cs:line 171
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding)
   at System.IO.File.ReadAllText(System.String path, System.Text.Encoding encoding) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 542
   at System.IO.File.ReadAllText(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 537
   at app.MismatchedFileNames() in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 45
   at app.RunTest(System.String banner, System.Action test) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 24
   at app.Main(System.String[] args) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 39

File 1: Hello, world - test2.txt here


-=-=-=-=-=-=- MONO_IOMAP REPORT -=-=-=-=-=-=-
 - Requested file path: 'files/test.txt'
 -     Found file path: 'Files/TesT.txt'

-= Stack Trace =-
   at System.Environment.get_StackTrace() in /usr/src/tmp/mono/mcs/class/corlib/System/Environment.cs:line 183
   at System.IO.MonoIO.GetFileAttributes(System.String , MonoIOError ByRef )
   at System.IO.MonoIO.ExistsDirectory(System.String path, MonoIOError ByRef error) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/MonoIO.cs:line 252
   at System.IO.Directory.Exists(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/Directory.cs:line 201
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/FileStream.cs:line 246
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 331
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/StreamReader.cs:line 171
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding)
   at System.IO.File.ReadAllText(System.String path, System.Text.Encoding encoding) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 542
   at System.IO.File.ReadAllText(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 537
   at app.MismatchedFileNames() in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 46
   at app.RunTest(System.String banner, System.Action test) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 24
   at app.Main(System.String[] args) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 39

File 2: Hello, world - TesT here


Running test: Access from an assembly.
Reading a file.
-=-=-=-=-=-=- MONO_IOMAP REPORT -=-=-=-=-=-=-
 - Requested file path: 'fileS/tESt.txt'
 -     Found file path: 'Files/TesT.txt'

-= Stack Trace =-
   at System.Environment.get_StackTrace() in /usr/src/tmp/mono/mcs/class/corlib/System/Environment.cs:line 183
   at System.IO.MonoIO.GetFileAttributes(System.String , MonoIOError ByRef )
   at System.IO.MonoIO.ExistsDirectory(System.String path, MonoIOError ByRef error) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/MonoIO.cs:line 252
   at System.IO.Directory.Exists(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/Directory.cs:line 201
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/FileStream.cs:line 246
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 331
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/StreamReader.cs:line 171
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding)
   at System.IO.File.ReadAllText(System.String path, System.Text.Encoding encoding) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 542
   at System.IO.File.ReadAllText(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 537
   at LibClass.ReadFile(System.String dir, System.String fileName) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-lib.cs:line 12
   at app.AccessFromAssembly() in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 51
   at app.RunTest(System.String banner, System.Action test) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 24
   at app.Main(System.String[] args) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 40

-=-=-=-=-=-=- MONO_IOMAP REPORT -=-=-=-=-=-=-
 - Requested file path: '/home/grendel/Projects/work/iomap-report-sample/fileS'
 -     Found file path: '/home/grendel/Projects/work/iomap-report-sample/Files'

-= Stack Trace =-
   at System.Environment.get_StackTrace() in /usr/src/tmp/mono/mcs/class/corlib/System/Environment.cs:line 183
   at System.IO.MonoIO.GetFileAttributes(System.String , MonoIOError ByRef )
   at System.IO.MonoIO.ExistsDirectory(System.String path, MonoIOError ByRef error) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/MonoIO.cs:line 252
   at System.IO.Directory.Exists(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/Directory.cs:line 201
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/FileStream.cs:line 270
   at System.IO.FileStream..ctor(System.String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 331
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/StreamReader.cs:line 171
   at System.IO.StreamReader..ctor(System.String path, System.Text.Encoding encoding)
   at System.IO.File.ReadAllText(System.String path, System.Text.Encoding encoding) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 542
   at System.IO.File.ReadAllText(System.String path) in /usr/src/tmp/mono/mcs/class/corlib/System.IO/File.cs:line 537
   at LibClass.ReadFile(System.String dir, System.String fileName) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-lib.cs:line 12
   at app.AccessFromAssembly() in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 51
   at app.RunTest(System.String banner, System.Action test) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 24
   at app.Main(System.String[] args) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:line 40

Reading file: Hello, world - TesT here



-=-=-=-=-=-=-= MONO_IOMAP Stats -=-=-=-=-=-=-=
    Count: 2
Requested: /home/grendel/Projects/work/iomap-report-sample/fileS
   Actual: /home/grendel/Projects/work/iomap-report-sample/Files
Locations:
        LibClass:ReadFile (string,string) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-lib.cs:12

    Count: 7
Requested: fileS/tESt.txt
   Actual: Files/TesT.txt
Locations:
        LibClass:ReadFile (string,string) in /home/grendel/Projects/work/iomap-report-sample/iomap-report-lib.cs:12

    Count: 4
Requested: /home/grendel/Projects/work/iomap-report-sample/files
   Actual: /home/grendel/Projects/work/iomap-report-sample/Files
Locations:
        app:MismatchedFileNames () in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:45
        app:MismatchedFileNames () in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:45

    Count: 7
Requested: files/Test2.txt
   Actual: Files/test2.txt
Locations:
        app:MismatchedFileNames () in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:45

    Count: 7
Requested: files/test.txt
   Actual: Files/TesT.txt
Locations:
        app:MismatchedFileNames () in /home/grendel/Projects/work/iomap-report-sample/iomap-report-sample.cs:46

 

Tags:

Mono | ASP.NET | Tip

Tip: Mono ASP.NET application burning CPU in idle state - FileSystemWatcher

by grendel 13. May 2009 10:40

Update: added sample code to detect the watcher in use, courtesy of Robert Jordan - thanks!

mod_mono is an Apache module for hosting ASP.NET applications. The module itself doesn't run any .NET code, instead it spawns a backend server (mod-mono-server.exe for ASP.NET 1.1 and mod-mono-server2.exe for ASP.NET 2.0) which is handed all the requests coming in from the client browser and sends back response generated by the application.

If you run Mono on a VPS server (e.g. Xen, OpenVZ) then you don't usually have any control over what Linux kernel version and with what capabilities you run. It may happen that the kernel lacks capabilities used by parts of the Mono runtime and Mono will have to fall back to other methods of doing the same task. One such part is the FileSystemWatcher class which is used to monitor changes to files/directories on disk so that the application can take any steps it deems necessary in reaction to file creation/deletion/modification events.

Mono's FileSystemWatcher does its best to perform its assigned task in various environments, under various operating systems. Part of the effort is selecting the actual filesystem monitoring backend best for the runtime environment. Under Unix the supported backends are as follow:

  • FAM
  • kevent (BSD*/MacOSX only)
  • gamin
  • inotify (Linux only)
  • Managed watcher
Out of those, assuming you run Linux, inotify is the preferred backend mechanism as it requires no polling effort on userland application part, instead the Linux kernel will notify the application (in our case the Mono runtime) whenever interesting events happen. However, it requires the Linux kernel to support the mechanism and, what's more important, for your VPS operator to actually include the support in the kernel your VPS runs on.

If your kernel doesn't support inotify, Mono will attempt to use FAM and Gamin which are userland daemons doing active filesystem polling but outside of the consumer application. The consumer application will use provided FAM/Gamin libraries to receive events and react to them. Performance of this setup is worse than inotify but not tragic.

Should Mono fail to detect inotify, FAM or Gamin support, it will fall back to the last resort option - the managed watcher. This watcher is implemented in managed code and uses a separate thread for filesystem monitoring, polling for changes on selected files/directories. As the application may (and in the case of ASP.NET sometimes does) watch directories recursively, it might be a very expensive situation requiring checking changes to a big set of files. Each change detection run requires checking whether a file/directory exists (in case of the Managed watcher those are two stat (2) calls) and then checking the file metadata for changes (size, modification times etc) and, possibly, generating an event. This happens approximately every 750ms and can generate substantial load on the server's CPU.

If you notice (using top or htop applications) that your copy of mod-mono-server burns several per-cent of CPU but is otherwise in the S (Sleeping) process state, chances are your application is using the managed watcher. You can confirm that by using htop which allows you to watch individual process threads - you will see two threads consuming nearly the same amount of CPU time and one of them waking up every ~750ms.

The cure for the itch is easy, if you can live without filesystem monitoring (that means your application will not auto-restart when you modify Web.config, files won't be recompiled if you modify a code-behind .cs or an .aspx, .ascx etc. files). Mono supports a MONO_MANAGED_WATCHER environment variable which can be set to value disable with the effect of definitely disabling filesystem monitoring (it will use a "dumb" implementation of the watcher backend which does nothing) and relieve your application of the filesystem polling chores described above.

You can set the environment variable for your Apache VirtualHost by using the following mod_mono directive:

MonoSetEnv [server_alias] MONO_MANAGED_WATCHER=disable

Sample program to detect which watcher backend is used:

using System;
using System.Reflection;
using System.IO;

class Program {

	public static void Main()
	{
		object watcher = new FileSystemWatcher()
			.GetType ()
			.GetField ("watcher", BindingFlags.NonPublic | BindingFlags.Static)
			.GetValue (null);
		
		Console.WriteLine (watcher != null
				   ? watcher.GetType ().FullName
				   : "unknown");
	}
}

Tags:

Mono | mod_mono | Tip

RecentComments

Comment RSS