NovelEssay.com Programming Blog

Exploration of Big Data, Machine Learning, Natural Language Processing, and other fun problems.

Avoid Deadlocks when Reading StandardOutput from a Process in C#

Let's start with some C# code that kicks off a new Process like this:

System.Diagnostics.Process p = System.Diagnostics.Process.Start(info);
p.WaitForExit();
Console.WriteLine(p.StandardOutput.ReadToEnd());

The process will do some Console.Write, and we wan't to capture that. In order to capture the child Process' output, we'll read StandardOutput. That will often work great... until it doesn't.


This can cause deadlocks where your parent process is stuck waiting due to a variety of scenarios.

  1. The child process never exits
  2. The child process writes so much data that it needs to be consumed with a stream pattern.

A good pattern is to set up event handlers on OutputDataReceived. After you append your event handler, then you need to call BeginOutputReadLine

using (AutoResetEvent outputWaitHandle = new AutoResetEvent(false))
using (AutoResetEvent errorWaitHandle = new AutoResetEvent(false))
{
using (Process process = new Process())
{
process.StartInfo.FileName = filename;
process.StartInfo.Arguments = arguments;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
StringBuilder output = new StringBuilder();
StringBuilder error = new StringBuilder();

process.OutputDataReceived += (sender, e) => {
if (e.Data == null)
{
outputWaitHandle.Set();
}
else
{
output.AppendLine(e.Data);
}
};
process.ErrorDataReceived += (sender, e) =>
{
if (e.Data == null)
{
errorWaitHandle.Set();
}
else
{
error.AppendLine(e.Data);
}
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
if (process.WaitForExit(timeout) &&
outputWaitHandle.WaitOne(timeout) &&
errorWaitHandle.WaitOne(timeout))
{
// Your process finished and check process.ExitCode now.
}
else
{
// Your process timed out.
}
}
}


This pattern is also very good at setting up a non-infinite wait time for your child process. Many other solution patterns for managing child processes assume an infinite wait, and that is often not a good choice for many of these advanced needs.





















Converting a text string to a Brushes Color object in C#

We want to dynamically go from a list of Colors (ie, Red, Yellow, White) to the Brush object types shown in my previous article: Adding Text to Images with C# .Net Bitmap objects - We'll show a few solutions for mapping strings to Brush Color objects.


Try this one first. Simply map your string to a Color using the FromName function like this:

Color yellowColor = Color.FromName("Yellow");

Alternatively, you can use a ColorConverter like this:

TypeConverter typeConverter1 = TypeDescriptor.GetConverter(typeof(Color));
TypeConverter typeConverter2 = new ColorConverter();
Color yellowColor2 = (Color)tc.ConvertFromString("Yellow");

Lastly, you can use Reflection on Color or Brush like this:

Color yellowColor3 = (Color)typeof(Color).GetProperty("Yellow").GetValue(null, null);



Adding Text to Images with C# .Net Bitmap objects

This article will show you some examples of how to add text to images with C# .Net Bitmap objects. This works for jpg, png, bitmap, and other image format types.

Procedure Overview:

  1. Create a Bitmap object with your source image.
  2. Create a RectangleF object around your source image.
  3. Create a Graphics object using your source Bitmap object
  4. Set several configuration values on your Graphics object that make the text look better in most cases.
  5. Draw your text string to the rectangle with all of the specified settings.
  6. Flush the changes and save your final output.

Here's some example code that implements the above procedure:

// Load the original image. Can be jpg, png, bmp, etc...
Bitmap bmp = new Bitmap("myImage.jpg");
// Create a rectangle for the entire bitmap
RectangleF rectf = new RectangleF(0, 0, bmp.Width, bmp.Height);
// Create graphic object that will draw onto the bitmap
Graphics g = Graphics.FromImage(bmp);
// ------------------------------------------
// Ensure the best possible quality rendering
// ------------------------------------------
// The smoothing mode specifies whether lines, curves, and the edges of filled areas use smoothing (also called antialiasing). One exception is that path gradient brushes do not obey the smoothing mode. Areas filled using a PathGradientBrush are rendered the same way (aliased) regardless of the SmoothingMode property.
g.SmoothingMode = SmoothingMode.AntiAlias;
// The interpolation mode determines how intermediate values between two endpoints are calculated.
g.InterpolationMode = InterpolationMode.HighQualityBicubic;
// Use this property to specify either higher quality, slower rendering, or lower quality, faster rendering of the contents of this Graphics object.
g.PixelOffsetMode = PixelOffsetMode.HighQuality;
// This one is important
g.TextRenderingHint = TextRenderingHint.AntiAliasGridFit;
// Create string formatting options (used for alignment)
StringFormat format = new StringFormat()
{
    Alignment = StringAlignment.Center,
    LineAlignment = StringAlignment.Center
};
// Draw the text onto the image
g.DrawString("Visit StyleMyImage.com", new Font("Tahoma",8), Brushes.Black, rectf, format);
// Flush all graphics changes to the bitmap
g.Flush();
// Now save or use the bitmap
image.Image = bmp;

The following are common items you may want to customize: Fonts, Size, Color, Text Position, etc...


If you want to change your font type or font size, edit the values you set in this part of the code:

new Font("Tahoma",14)

If you want the text to be Yellow, change the 

Brushes.Black 

to 

Brushes.Yellow


If you want the text to be in the bottom right corner, change the Alignment values in the StringFormat object.

StringFormat format = new StringFormat()
{
Alignment = StringAlignment.Far,
LineAlignment = StringAlignment.Far
};


Finally, if you want to change the Text drawn on to the image, change the first argument passed to g.DrawString from Visit StyleMyImage.com to whatever you'd like it to say.


Tesseract 4.0 C# .Net Wrapper Released!

This article is about the Tesseract 4.0 C# .Net Wrapper that is only a few days old as of April 2017.


You are probably familiar with the Tesseract 3.04 C# .Net Wrapper found here:

https://github.com/charlesw/tesseract

That is already available as a Nuget package and has many downloads.


Just about a week ago, an Alpha release of the Tesseract 4.0 C# .Net wrapper was published here:

https://github.com/tdhintz/tesseract4win64

This is an x64 only .Net assembly. 


Find the Tesseract 4.0 language packs here:

https://github.com/tesseract-ocr/tessdata

When I load English only language pack, it uses a reasonable 180MB of RAM. I tried to load "all languages", and it was using over 8GB of RAM. 


This build is incredibly slow for debug mode. It runs 5-8X slower in debug mode than release mode, so watch out for that.


Amazingly, the .Net wrapper API works exactly the same as the Tesseract C# .Net 3.0 wrapper! (When you read about how the engine changed a huge amount and using LTSM networks, this will be more amazing to you.)


A very simple usage example works like this:

var tessEngine = new TesseractEngine(tessdataPath, "eng");
using (Page page = tessEngine .Process(myImage))
{
    string resultText = page.GetText();


Be sure to drop these two files in your \bin\debug or \bin\release folder at a x64 sub-folder like this::

.\bin\release\x64\libtesseract400.dll
.\bin\release\x64\liblept1741.dll

When the Tesseract.dll 4.0 assembly loads, it needs to find those DLLs else it will throw an exception in your application.


There is a very nice Accuracy and Performance overview report of 3.04 versus 4.0 here:

https://github.com/tesseract-ocr/tesseract/wiki/4.0-Accuracy-and-Performance

I agree with it's findings generally, but my own personal tests are not nearly as "improved" versus 3.04. I have a regression test that contains about 2200 pages, and I'm observing plenty of slower and less precise OCR results with Tesseract 4.0. It is certainly not all "better and faster" as of April 2017. Since this is an extremely new Alpha release, I have high hopes that it will improve over time.


Make a Slack WebHook plugin with C# .Net, Nancy, and Ngrok.

This blog article will walk you through making a C# webhook plugin for Slack. We'll be using Nancy to setup a small web service, and ngrok to expose our service publicly so Slack can call it.


First, make a new C# console application with Visual Studio, and install a few nuget packages. The interesting packages are Nancy, Nancy.Hosting.Self, and Slack.Webhooks.


Next, we need to create a NancyHost and start it up. You'll want your console application's main to look roughly like this:

        static void Main()
        {
            JsConfig.EmitLowercaseUnderscoreNames = true;
            JsConfig.IncludeNullValues = false;
            JsConfig.PropertyConvention = JsonPropertyConvention.Lenient;
            using (var host = new NancyHost(new Uri("http://localhost:1234")))
            {
                host.Start();
                Console.ReadLine();
            }
            return;
        }

When our application runs, it launches a service that listens on localhost:1234 for requests.


Of course, you'll need some using statements like this:

using Nancy;
using Nancy.Hosting.Self;
using Nancy.ModelBinding;
using Newtonsoft.Json;
using ServiceStack.Text;
using Slack.Webhooks;


Since our application will be listening on localhost:1234, we need to add request handlers. For our slack webhook plugin examlpe, we just need to handle a post. We'll create a WebhookModule class that inherrits from NancyModule, and has a Post handler like this code:

public class WebhookModule : NancyModule
{
public WebhookModule()
{
Post["/"] = _ =>
{
var model = this.Bind<HookMessage>();
var message = string.Empty;
SlackAttachment attachment = null;
message = string.Format("@{0} Hello", model.UserName);
if (!string.IsNullOrWhiteSpace(message))
{
SlackMessage sm = new SlackMessage { Text = message, Username = "MyChat.Bot.Greeting", IconEmoji = Emoji.Ghost };
if(attachment != null)
{
sm.Attachments = new List<SlackAttachment>();
sm.Attachments.Add(attachment);
}
return sm;
}
return null;
};
}
}

When Post is received, this will receive a HookMessage and respond with a "Hello User" message. That response will be received by Slack and should output the "Hello User" message to your slack chat channel.


Here are a few other classes you'll need that define the HookMessage and some other Nancy boiler plate configuration:

    public class HookMessage
    {
        public string Token { get; set; }
        public string TeamId { get; set; }
        public string ChannelId { get; set; }
        public string ChannelName { get; set; }
        public string UserId { get; set; }
        public string UserName { get; set; }
        public string Text { get; set; }
        public string TriggerWord { get; set; }
    }
    public class TitleCaseFieldNameConverter : IFieldNameConverter
    {
        public string Convert(string fieldName)
        {
            return fieldName.ToTitleCase();
        }
    }
    public class Bootstrapper : DefaultNancyBootstrapper
    {
        protected override void ApplicationStartup(Nancy.TinyIoc.TinyIoCContainer container, Nancy.Bootstrapper.IPipelines pipelines)
        {
            container.Register<IFieldNameConverter, TitleCaseFieldNameConverter>();
            base.ApplicationStartup(container, pipelines);
        }
    }


At this point, you should be able to run your C# console application and be listening for Post requests on localhost:1234. Next, follow my ngrok block article to setup ngrok to expose your localhost:1234 service to a public address:

http://blog.novelessay.com/post/make-local-mysql-instance-publicly-available-for-a-mvc-net-website-with-ngrok

When your ngrok is ready to run, you can make your service publicly available by starting the ngrok service like this:

ngrok.exe http 1234


Lastly, you'll need to go in to your Slack configuration, and setup an Outgoing Webhook. Look in the "Browse Apps" -> "Custom Integrations" -> "Outgoing WebHooks" section. You will probably discover it more tricky to find where it was than to actually configure it.



You need to configure which channels you want your webhook to interact with, set the ngrok address that your service is serving on, and provide a token that Slack will send. You should update your C# console application to check the token value, but that's not entirely necessary for this system to work.


Here's an example of how I have an Outgoing WebHook configured in Slack:



That's everything! Give your Slack Bot a try. 

C# Bitmap Image Scaling for Bmp, Jpg, and Png

How do you scale your bmp, jpg, or png images in C#? Use System.Drawing.BitMap and follow the instructions in this article.


(Be sure to wrap all of your MemoryStreams and Bitmap objects with Using statements, so they get disposed, otherwise you'll have some memory leaks and locks on files.)


First, add references to System.Drawing in your C# Visual Studio project, and you'll want a using statement near the top of your code too:

using System.Drawing;

Then, we'll load up our image in to a Bitmap object and save it to a memory stream like this:

string sourceFile = "c:\my.png";

using (Bitmap srcBitmap = new Bitmap(sourceFile))
using (MemoryStream streamSrc = new MemoryStream())
{
srcBitmap.Save(streamSrc, System.Drawing.Imaging.ImageFormat.Jpeg);
We will calculate the area of the source image using the height and width attributes:
	int sourceImageArea = srcBitmap.Width * srcBitmap.Height;


If the source image area exceeds a threshold, we'll calculate a scale factor that we need to reduce the height and width by. That new scale factor will be applied to get a new width and height.

		float scaleRatio = maxOutputPixelArea;
scaleRatio /= sourceImageArea; // scaleRatio is less than 1
float newW = srcBitmap.Width;
newW *= scaleRatio;
float newH = srcBitmap.Height;
newH *= scaleRatio;

Lastly, we'll create a new Bitmap to save the converted image to with our new dimensions:

		using (Bitmap resizedImg = ResizeImage(srcBitmap, Convert.ToInt32(newW), Convert.ToInt32(newH)))
using (MemoryStream resizedImgStrm = new MemoryStream())
{
resizedImg.Save(resizedImgStrm, System.Drawing.Imaging.ImageFormat.Jpeg);
imageBytes = resizedImgStrm.ToArray();
// Save imageBytes to file, store to Database, or whatever
}


The full code looks like this:

string sourceFile = "c:\my.png";
int maxOutputPixelArea = 3000000;
using (Bitmap srcBitmap = new Bitmap(sourceFile))
using (MemoryStream streamSrc = new MemoryStream())
{
// Convert the image to byte[]
srcBitmap.Save(streamSrc, System.Drawing.Imaging.ImageFormat.Jpeg);
int sourceImageArea = srcBitmap.Width * srcBitmap.Height;
if (sourceImageArea > maxOutputPixelArea)
{
float scaleRatio = maxOutputPixelArea;
scaleRatio /= sourceImageArea; // scaleRatio is less than 1
float newW = srcBitmap.Width;
newW *= scaleRatio;
float newH = srcBitmap.Height;
newH *= scaleRatio;
using (Bitmap resizedImg = ResizeImage(srcBitmap, Convert.ToInt32(newW), Convert.ToInt32(newH)))
using (MemoryStream resizedImgStrm = new MemoryStream())
{
resizedImg.Save(resizedImgStrm, System.Drawing.Imaging.ImageFormat.Jpeg);
imageBytes = resizedImgStrm.ToArray();
// Save imageBytes to file, store to Database, or whatever
}
}


Finally, the ResizeImage code is this:

private static Bitmap ResizeImage(Image image, int width, int height)
{
var destRect = new Rectangle(0, 0, width, height);
var destImage = new Bitmap(width, height);
destImage.SetResolution(image.HorizontalResolution, image.VerticalResolution);
using (var graphics = Graphics.FromImage(destImage))
{
graphics.CompositingMode = CompositingMode.SourceCopy;
graphics.CompositingQuality = CompositingQuality.HighQuality;
graphics.InterpolationMode = InterpolationMode.HighQualityBicubic;
graphics.SmoothingMode = SmoothingMode.HighQuality;
graphics.PixelOffsetMode = PixelOffsetMode.HighQuality;
using (var wrapMode = new ImageAttributes())
{
wrapMode.SetWrapMode(WrapMode.TileFlipXY);
graphics.DrawImage(image, destRect, 0, 0, image.Width, image.Height, GraphicsUnit.Pixel, wrapMode);
}
}
return destImage;
}

Social Like and Share Buttons with AngularJS: 'The Angular Way'

You're using AngularJS for your web site, and you want to add some social media share and like buttons that look roughly like this:



Here's what you should do - Setup your App to inject the angulike like this:

(Full Angulike code is near the bottom of this article.)


var myApp = angular.module('myApp', ['angulike'])
.run([
      '$rootScope', function ($rootScope) {
          $rootScope.facebookAppId = 'Your FB App Id Here'; // set your facebook app id here
      }
])

Be sure to buzz over to Facebook.com and obtain an App ID, and set it in the code above.


Then, in your controller code, create a scope variable that has your Url, Name, and an Imgae to be shared:

    $scope.myModel = {
        Url: 'http://blog.novelessay.com',
        Name: "blog.novelessay.com makes you smarter!",
        ImageUrl: 'http://blog.novelessay.com/Images/awesome.jpg'
    };


In your HTML, simply add a few divs like this:

<div fb-like="myModel.Url"></div>
<div tweet="myModel.Name" tweet-url="myModel.Url"></div>
<div google-plus="myModel.Url"></div>
<div pin-it="myModel.Name" data-pin-it-image="myModel.ImageUrl" pin-it-url="myModel.Url"></div>

They will each become the various like and share buttons. Feel free to add css class styling to the div to fit your needs.

Of course, we need to add the AngularJs and Angulike.js script tags to our HTML too:

    <script src="https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.5.5/angular.min.js"></script>
    <script type='text/javascript' src='/scripts/angulike.js'></script>

And, finally, Angulike.js is the following:

/**
 * AngularJS directives for social sharing buttons - Facebook Like, Google+, Twitter and Pinterest 
 * @author Jason Watmore <jason@pointblankdevelopment.com.au> (http://jasonwatmore.com)
 * @version 1.2.0
 */
(function () {
    angular.module('angulike', [])

      .directive('fbLike', [
          '$window', '$rootScope', function ($window, $rootScope) {
              return {
                  restrict: 'A',
                  scope: {
                      fbLike: '=?'
                  },
                  link: function (scope, element, attrs) {
                      if (!$window.FB) {
                          // Load Facebook SDK if not already loaded
                          $.getScript('//connect.facebook.net/en_US/sdk.js', function () {
                              $window.FB.init({
                                  appId: $rootScope.facebookAppId,
                                  xfbml: true,
                                  version: 'v2.0'
                              });
                              renderLikeButton();
                          });
                      } else {
                          renderLikeButton();
                      }

                      var watchAdded = false;
                      function renderLikeButton() {
                          if (!!attrs.fbLike && !scope.fbLike && !watchAdded) {
                              // wait for data if it hasn't loaded yet
                              watchAdded = true;
                              var unbindWatch = scope.$watch('fbLike', function (newValue, oldValue) {
                                  if (newValue) {
                                      renderLikeButton();

                                      // only need to run once
                                      unbindWatch();
                                  }

                              });
                              return;
                          } else {
                              element.html('<div class="fb-like"' + (!!scope.fbLike ? ' data-href="' + scope.fbLike + '"' : '') + ' data-layout="button_count" data-action="like" data-show-faces="true" data-share="true"></div>');
                              $window.FB.XFBML.parse(element.parent()[0]);
                          }
                      }
                  }
              };
          }
      ])

      .directive('googlePlus', [
          '$window', function ($window) {
              return {
                  restrict: 'A',
                  scope: {
                      googlePlus: '=?'
                  },
                  link: function (scope, element, attrs) {
                      if (!$window.gapi) {
                          // Load Google SDK if not already loaded
                          $.getScript('//apis.google.com/js/platform.js', function () {
                              renderPlusButton();
                          });
                      } else {
                          renderPlusButton();
                      }

                      var watchAdded = false;
                      function renderPlusButton() {
                          if (!!attrs.googlePlus && !scope.googlePlus && !watchAdded) {
                              // wait for data if it hasn't loaded yet
                              watchAdded = true;
                              var unbindWatch = scope.$watch('googlePlus', function (newValue, oldValue) {
                                  if (newValue) {
                                      renderPlusButton();

                                      // only need to run once
                                      unbindWatch();
                                  }

                              });
                              return;
                          } else {
                              element.html('<div class="g-plusone"' + (!!scope.googlePlus ? ' data-href="' + scope.googlePlus + '"' : '') + ' data-size="medium"></div>');
                              $window.gapi.plusone.go(element.parent()[0]);
                          }
                      }
                  }
              };
          }
      ])

      .directive('tweet', [
          '$window', '$location',
          function ($window, $location) {
              return {
                  restrict: 'A',
                  scope: {
                      tweet: '=',
                      tweetUrl: '='
                  },
                  link: function (scope, element, attrs) {
                      if (!$window.twttr) {
                          // Load Twitter SDK if not already loaded
                          $.getScript('//platform.twitter.com/widgets.js', function () {
                              renderTweetButton();
                          });
                      } else {
                          renderTweetButton();
                      }

                      var watchAdded = false;
                      function renderTweetButton() {
                          if (!scope.tweet && !watchAdded) {
                              // wait for data if it hasn't loaded yet
                              watchAdded = true;
                              var unbindWatch = scope.$watch('tweet', function (newValue, oldValue) {
                                  if (newValue) {
                                      renderTweetButton();

                                      // only need to run once
                                      unbindWatch();
                                  }
                              });
                              return;
                          } else {
                              element.html('<a href="https://twitter.com/share" class="twitter-share-button" data-text="' + scope.tweet + '" data-url="' + (scope.tweetUrl || $location.absUrl()) + '">Tweet</a>');
                              $window.twttr.widgets.load(element.parent()[0]);
                          }
                      }
                  }
              };
          }
      ])

      .directive('pinIt', [
          '$window', '$location',
          function ($window, $location) {
              return {
                  restrict: 'A',
                  scope: {
                      pinIt: '=',
                      pinItImage: '=',
                      pinItUrl: '='
                  },
                  link: function (scope, element, attrs) {
                      if (!$window.parsePins) {
                          // Load Pinterest SDK if not already loaded
                          (function (d) {
                              var f = d.getElementsByTagName('SCRIPT')[0], p = d.createElement('SCRIPT');
                              p.type = 'text/javascript';
                              p.async = true;
                              p.src = '//assets.pinterest.com/js/pinit.js';
                              p['data-pin-build'] = 'parsePins';
                              p.onload = function () {
                                  if (!!$window.parsePins) {
                                      renderPinItButton();
                                  } else {
                                      setTimeout(p.onload, 100);
                                  }
                              };
                              f.parentNode.insertBefore(p, f);
                          }($window.document));
                      } else {
                          renderPinItButton();
                      }

                      var watchAdded = false;
                      function renderPinItButton() {
                          if (!scope.pinIt && !watchAdded) {
                              // wait for data if it hasn't loaded yet
                              watchAdded = true;
                              var unbindWatch = scope.$watch('pinIt', function (newValue, oldValue) {
                                  if (newValue) {
                                      renderPinItButton();

                                      // only need to run once
                                      unbindWatch();
                                  }
                              });
                              return;
                          } else {
                              element.html('<a href="//www.pinterest.com/pin/create/button/?url=' + (scope.pinItUrl || $location.absUrl()) + '&media=' + scope.pinItImage + '&description=' + scope.pinIt + '" data-pin-do="buttonPin" data-pin-config="beside"></a>');
                              $window.parsePins(element.parent()[0]);
                          }
                      }
                  }
              };
          }
      ]);

})();

A big thanks to Jason Watmore the author of Angulike for making the base set of Angular friendly code for this scenario.


After following this process, you should get some social buttons to like and share your content that look like this:



Installing Python Chainer and Theano on Windows with Anaconda for GPU Processing

Let's say you want to do some GPU processing on Windows and you want to use Python, because of awesome things like this:


We'll show the setup steps for installing Python Chainer and Theano on Windows 10 in this blog article.


Some Terms:

CUDAan API model created by Nvidia for GPU processing.

cuDNN - a neural network plugin library for CUDA

Chainer - a Python neural network framework package

Theano - a Python deep learning package


Initial Hardware and OS Requirements:

You need an Nvidia CUDA supported video card. (I have a NVidia GeForce GTX 750 Ti.) Check for your GPU card in the support list found here: https://developer.nvidia.com/cuda-gpus 

You need Windows 10. (Everything in this procedure is x64.)


Important: 

Versions matter a lot. I tried to do this exact same setup with Python 2.7, and I was not successful. I tried to do the same thing with Anaconda 2, and that didn't work. I tried to do this same thing with cuDNN 5.5, and that didn't work. - So many combinations didn't work for me that I decided to write about what did work.


Procedure:

1) Install Visual Studio 2015. You must install Visual Studio before installing the CUDA tool kit. You need the \bin\cl.exe compiler. I have the VS2015 Enterprise Edition, but the VS2015 Community Edition is free here: https://www.microsoft.com/en-us/download/details.aspx?id=48146


2) Install the CUDA Tool kit found here: https://developer.nvidia.com/cuda-downloads

That installs v8.0 to a path like this: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0


3) Download the cuDNN v5.0 here: https://developer.nvidia.com/cudnn

There is a v5.1 there, but it did not work for me. Feel free to try it, but I suggest trying v5.0 first.

The cuDNN is just 3 files. You'll want to drop them in the CUDA path:

  • Drop the cudnn.h file in the folder:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\
  • Drop the cudnn64_5.dll file in the folder:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin\
  • Drop the cudnn.lib file in the folder:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64\

4) Install Anaconda 3.6 for Windows x64 found here: https://repo.continuum.io/archive/Anaconda3-4.3.1-Windows-x86_64.exe

In case that link breaks, this is the page I found it at: https://www.continuum.io/downloads

You'll be doing most of your Anaconda/Python work in the Anaconda Console window. If Windows does not give you a nice link to the Anaconda Console, make a short cut with a link that looks like this:

"%windir%\system32\cmd.exe " "/K" C:\ProgramData\Anaconda3\Scripts\activate.bat C:\ProgramData\Anaconda3

I installed Anaconda for "All Users", so it put it at ProgramData. If you install to just one user, it puts Anaconda at a c:\users\<your name>\ path.


5) Building python packages requires a gcc/g++ compiler. Install MinGW for x64 here: https://sourceforge.net/projects/mingw-w64/

WARNING: During this install, be sure to pick the x86_64 install and not the i686 install!

The default install for MinGW is at c:\Program Files\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin

The space in Program Files will break stuff later, so move it to something like this instead:

C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin


6) Environment paths! 

If you have no idea how to set Enviornment variables in Windows, here's a link that describes how to do that: http://www.computerhope.com/issues/ch000549.htm

Add a variable called "CFlags" with this value:

  • -IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include

Add a variable called "CUDA_PATH" with this value:

  • C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0

Add a variable called "LD_LIBRARY_PATH" with this value:

  • C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64

Add a variable called "LDFLAGS" with this value:

  • -LC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64

Add all of the following to your PATH variable (or ensure they exist):

  • C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin
  • C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin
  • C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\libnvvp
  • C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin
  • C:\ProgramData\Anaconda3
  • C:\ProgramData\Anaconda3\Scripts
  • C:\ProgramData\Anaconda3\Library\bin

(The Anacond3 paths might get set automatically for you.)


7) Next, bring up your Anaconda console prompt and install some packages. Type the following lines:

pip install Pillow
pip install Pycuda
pip install Theano
pip install Chainer
If any of those fail to install, stop and figure out why. Append a -vvvv to the end of the install lines to get a very-very-very verbose dump of the install process. 

Note: If you can't get pycuda to install due to "missing stdlib.h" errors, you can get the pycuda Whl file and install that directly instead.

It likely is because one of your steps #1-6 isn't quite right, or because your GCC compiler is trying to use an old x32 version that you installed long ago. (That was the case for me. I had Cygwin and a x32 GCC compiler that caused failing pip package installs.)

I also had some build fails on Chainer with some errors about "_hypot" being undefined. I fixed those by going to C:\ProgramData\Anaconda3\include\pyconfig.h, and commenting out the two places in that file that do this:
//#define hypot _hypot
That appear to have fixed that issue for me, but there's probably a better solution.

8) Sanity checks and smoke tests:
First, try to import the packages from a python command window. You can run this directly from your Anaconda console like this:
  • python -c "import theano"
  • python -c "import chainer"
  • python -c "import cupy"
If one of them fails, identify the error message and ask the Google about it. They should all work:


A last smoke test is to get the "Hello GPU" test code from here:

Here's a copy:
import pycuda.autoinit
import pycuda.driver as drv
import numpy
from pycuda.compiler import SourceModule
mod = SourceModule("""
__global__ void multiply_them(float *dest, float *a, float *b)
{
  const int i = threadIdx.x;
  dest[i] = a[i] * b[i];
}
""")
multiply_them = mod.get_function("multiply_them")
a = numpy.random.randn(400).astype(numpy.float32)
b = numpy.random.randn(400).astype(numpy.float32)
dest = numpy.zeros_like(a)
multiply_them(
        drv.Out(dest), drv.In(a), drv.In(b),
        block=(400,1,1), grid=(1,1))
print (dest-a*b)

I had to change the last line of that code to have parenthesis around it like this:
print (dest-a*b)

When you run that with a command like this:
python pycuda_test.py
You should get an output of 0's that look like this:



Conclusion:
If you've gotten to here, congratulations! Your Windows 10 environment should be all setup to run Python GPU processing.

My GPU has been running for days at 90%, and my CPU is free for other work. This was a seriously miserable to figure out, but now it feels like my computer doubled the processing power!

Enjoy the awesome:

Handling inconsistent Url formats in C#

Let's say you have a whole bunch of Urls in a database or whatever collection format, but some are like these examples:

  • NovelEssay.com
  • www.NovelEssay.com
  • http://NovelEssay.com
  • http://www.NovelEssay.com

Is there an easy way to handle these variety of formats in C#?


Yes, we'll do some prefix detection and then use a Uri class to help parse the data.


First, we assume our original data is in the HomePage variable on the Record object. We'll detect if HomePage starts with "http://". If it doesn't, we'll add that:

if (!Record.HomePage.StartsWith(HTTP_PREFIX))
{
Record.HomePage = HTTP_PREFIX + Record.HomePage;
}

Next, we'll try to parse the HomePage with the Uri class like this:

try
{
    Uri myUri = new Uri(Record.HomePage);
}
catch (Exception e)
{
    // HomePage is not parsable as Uri object
}

If HomePage is still in a bad format, the Uri constructor will throw an exception that we want to catch.


Finally, if an exception isn't thrown, we can use the new myUri to extract the base domain and other parts of the Uri like this:
Record.Domain = myUri.Host;
One last trick, is to check the Record.Domain or myUri.Host to see if it begins with a www. prefix. Depending on how you want to normalize your host information, you may want to add or remove the www prefix on the Host (or Domain) value.

That's all. Happy Uri parsing!