Friday, August 2, 2013

Password Hashing in .NET: The Right Way

Far too many times I see websites, including certain popular social media sites, that do not store passwords in a safe and secure manner. When you create a new account with a website and you get that confirmation e-mail that includes your login name and password, that is bad. Bad website! Bad!

Your password should NEVER be stored in a form that can be retrieved and sent to you in plain text, ever. This means it should never be encrypted in a form that can be decrypted. The sad reality of web security is that nothing is truly secure or uncrackable. If it can be done, it can be undone. A password should be stored in a hashed format, that is, a one-way encryption that can not be reversed.

Simply hashing a password is not enough though. A hashed password should be large enough that it cannot be susceptible to collision attacks. A collision attack is the equivalent to a brute-force attack on a hash value where the attacker tries to create a value that will result in the same has value when hashed. A hash value should also use a unique, cryptographically secure key, referred to as a "salt", to help generate the hash value. Unfortunately, even with all these steps, an attacker can still attempt to brute-force a hash collision. While we can't completely stop this from happening, we can slow the rate at which an attack can be made. In fact, we can slow the rate at which a collision attack can be down to a crawl, effectively mitigating the attack.

In this article, I will show you all of these steps. We'll briefly touch on hash algorithms and which are acceptable for password hashing, as well as those which are not. We'll cover salt use and briefly touch on entropy. We'll also cover iterative hashing and password-based key-derived functions, and validating hash values. Finally, we will create a fully-functional, production-ready example making use of all these features.

There are several hash algorithms that ship with the .NET framework out of the box and each has it's own ideal scenarios for use. Only certain algorithms should even be considered for hashing passwords though, since we need to ensure that our password hashes are of a large enough size that a collision attack would not be very effective. Below are some of the common hash algorithms and some details on each.

Name Password-Acceptable Details
MD5 no Produces 128-bit hash values which are not collision-resistant
SHA1 no Produces 160-bit hash values which have been found to be susceptible to attack
SHA2 yes Produces hash values in the sizes of 224-bit, 256-bit, 384-bit and 512-bit, which are immune to collision attacks as of yet

In the .NET framework, these hash algorithms have a few different implementations: CryptoServiceProvider versions, and Managed versions. The key difference between these two variations of each are that the CryptoServiceProvider implementations make use of the CryptoServiceProvider internally to compute the hash functions, while the Managed implementations make use of the managed library to provide their own implementation of the algorithm(s). The key difference between the two is that CryptoServiceProvider functions are in accordance with the NIST guidelines and therefor, approved for use in government software, where as the Managed functions do not adhere to those guidelines and are therefore not approved for use in government software.

There is one more hash algorithm worth mentioning here. The one we will be using in our example: Rfc2898DeriveBytes. This algorithm is in adherence to RFC standards for password storage and provides multiple benefits. It allows for hashing of variable length keys, implements iterative hashing and uses key-stretching. It also supports salting and is very strong against password-guessing attacks if used properly.

A password salt is a value used in addition to the password/passphrase to generate the hash value. It's like a password for a password. In the context of password storage, a salt value should be unique and not reused across multiple passwords. Each password should have it's own unique salt value. The most common reason otherwise secure authentication systems become breached are from static salt values. That is, using the same salt value for all passwords. In addition to being unique, a salt value should have a strong amount of entropy. Entropy is a measurement of randomness. For password storage, you should generate salt values with at least 128 random bits. In our example, we will be using the RNGCryptoServiceProvider to generate 128-bit cryptographically secure salt values.

Before we get started, I'd like to talk about the Rfc2898DeriveBytes algorithm we will be using. As I mentioned before, this performs iterative hashing, key stretching and allows for variable length key input. Sounds great but what does that all mean?

Variable key length means you can input any length of bytes and it will hash it, as opposed to some other algorithms where you must input data of a specific size. Variable length is ideal for passwords since passwords vary in length (or should).

Key stretching is a method of enhancing or strengthening the input value by stretching it's size and adding entropy (randomness to it). In this way, if a user was to create a relatively week password, it will be strengthened internally regardless. This is important because if one user was allowed to make a weak password, their password being cracked could lead to the entire user store having their passwords compromised as well. Unique salting helps mitigate this also.

Iterative hashing means the data is hashed repeatedly in a loop. This is also part of how key stretching takes place. By doing this you accomplish two things: The key is strengthened, and the speed at which a collision attack can take placed is decreased drastically. Ideally, you want the number of iterations to be an amount which does slow the speed but not to an extent that will affect the user experience. The suggested minimum value is 1'000 iterations, however many implementations will use much more than this, sometimes higher than 10'000. The amount which will be suitable without being noticeable to users depends on the hardware which it is running on.

Let's get started creating our password hashing class. We'll call the class "PBKDF2Managed" (in accordance with the Microsoft cryptography naming conventions).

namespace System.Security.Cryptography
{
    public sealed class PBKDF2Managed : IDisposable
    {

    }
}

We'll be inheriting from the IDisposable interface to allow for our class to be used in using (...) statements, although this isn't necessary as there is no unmanaged resources that must be disposed. This is simply a design preference in this case.

Next we're going to add some constants and fields to our class:

        const int SIGNATURE_SIZE = 4;
        const int SALT_SIZE = 128;
        const int KEY_SIZE = 256;
        const int HASH_SIZE = 1 + SIGNATURE_SIZE + SALT_SIZE + KEY_SIZE;

        int _iterations;
        byte[] _signature;
        byte[] _salt;
        byte[] _key;

The constants identify the static sizes of the various components of our hashed values. For performance reasons, we make them constant since their values will not be changing.

Next, we will add the following methods to the class:

        public byte[] ComputeHash(byte[] password)
        {
            if (password == null)
            {
                throw new ArgumentNullException("password");
            }
            using (Rfc2898DeriveBytes _pbkdf2 = new Rfc2898DeriveBytes(password, _salt, _iterations))
            {
                _key = _pbkdf2.GetBytes(KEY_SIZE);
            }
            byte[] _hash = new byte[HASH_SIZE];
            Buffer.BlockCopy(_signature, 0, _hash, 1, SIGNATURE_SIZE);
            Buffer.BlockCopy(_salt, 0, _hash, 1 + SIGNATURE_SIZE, SALT_SIZE);
            Buffer.BlockCopy(_key, 0, _hash, 1 + SIGNATURE_SIZE + SALT_SIZE, KEY_SIZE);
            return _hash;
        }

        public void Dispose()
        {
            // clean up any unmanaged resources
        }

The Dispose() method is simply to satisfy the inheritance of the IDisposable interface.

The ComputeHash method is what actually creates our hash values. First we check to see if the password byte array supplied is null, and throw a ArgumentNullException if it is. We don't want to try and hash a non-existent value. You may have also noticed that method accepts an array of bytes as the password input and not a string. We do this so that the method is "encoding-agnostic" so to speak. Instances of this class will never have to worry about what encoding the input was in, it simply handles the bytes of the data. This allows us to handle the encoding issues within the caller, also making this class localization-friendly.

Next we create an Rfc2898DeriveBytes instance in a using statement, supplying the password bytes, the salt, and the number of iterations to be performed. Within the using statement we assign the computed hash value of the data and assign it to the _key field. Don't be concerned about the initialization of the _signature, _key, _salt, and _iterations fields just yet. We take care of that in the constructors which we will be creating in the next step. Just know that this instance does all the heavy lifting for us and performs our iterative, key-stretching hash on our input using the values we provide it.

In the final stages of the ComputeHash() method, we are formatting and finalizing the output of our hashed value. We create a new byte array called _hash. This will hold the completed hash value returned to the caller. We use Buffer.BlockCopy() to write to this array as it is our best performance option here aside from using pointers. First, the _signature field's bytes are written to the hash array. Notice we are leaving the first byte of the hash array blank. We will be using this as a validation step in later use of our computed hashes, which we will cover later. The _signature byte array is actually an array of 4 bytes that represent the number of iterations used to compute the hash. This is used in later steps when we need to recreate the hash to validate password input against a stored password hash. Next, the _salt is written to the array immediately after the signature array. Like the _signature array, the _salt array is embedded in the hash so we can retrieve it for later use when validating user password input against stored password hashes. The reason we store the salt in the key itself is that it can be passed anywhere needed without being stored separately in a database. Databases can be compromised and if an attacked can simply look up the salt for a specific password, it gives them an advantage of already knowing which salt to use. Doing it our way, they will have to guess at which bytes in the array represent the salt value, which would be an extremely frustrating task, assuming they would even know it's embedded in it at all.

Finally, in the last step, the computed hash key is written to the hash byte array immediately after the salt value. Our hash value now contains a blank validation byte, a unique 128-bit salt, and a 256-bit computed hash value, giving us a 385-bit hash value in total.

Next we create the constructors for our class:

        public PBKDF2Managed(int iterations)
        {
            if (iterations < 1)
            {
                throw new CryptographicException("Invalid iteration value.");
            }
            _iterations = iterations;
            _signature = BitConverter.GetBytes(_iterations);
            _salt = new byte[SALT_SIZE];
            using (RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider())
            {
                _rng.GetBytes(_salt);
            }
        }

        public PBKDF2Managed(byte[] salt, int iterations)
        {
            if (iterations < 1)
            {
                throw new CryptographicException("Invalid iteration value.");
            }
            if (salt == null)
            {
                throw new CryptographicException("Invalid salt value.");
            }
            if (salt.Length != SALT_SIZE)
            {
                throw new CryptographicException("Invalid salt value.");
            }
            _iterations = iterations;
            _signature = BitConverter.GetBytes(_iterations);
            _salt = salt;
        }

        public PBKDF2Managed(byte[] hash)
        {
            if (hash == null)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            if (hash.Length != HASH_SIZE)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            if (hash[0] != 0x00)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            _signature = new byte[SIGNATURE_SIZE];
            Buffer.BlockCopy(hash, 1, _signature, 0, 4);
            _iterations = BitConverter.ToInt32(_signature, 0);
            if (_iterations < 1)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            _salt = new byte[SALT_SIZE];
            Buffer.BlockCopy(hash, 1 + SIGNATURE_SIZE, _salt, 0, SALT_SIZE);
        }

We have created 3 different initializers, and each one allows instances of our class to serve a different purpose, which are as follows:

The first initializer is used when we will be creating a password hash for a user. It accepts a 32-bit integer as its input parameter which tells the algorithm how many iterations are to be performed when hashing. Within the body of the constructor a secure, unique 128-bit salt is generated using the RNGCryptoServiceProvider I mentioned earlier. Next, the _signature byte array is assigned the byte-value of the iteration count. Also, note how the iterations parameter is validated to ensure the value is greater than or equal to 1. We don't want to iterate 0 times or -256 times, etc. Also note the type of exceptions we throw and the messages we use in them. They are vague, non-descriptive values. Most attackers find their way through vulnerabilities by using useful information found in overly informative exception messages. We don't want to give a potential attacker anything to use to their advantage.

The second constructor, like the first, will be used when creating a password hash for a user, however, this constructor assumes that a salt provided in the input parameter is to be used for computing the hash value. It will validate that the salt passed to it is the correct length and not null or it will throw non-descriptive error.

The third and final constructor is used when validating user input against a stored password hash. It accepts a hash value as it's input parameter, which is first validated to make sure that:

  1. the hash is not null
  2. the hash is of the correct length
  3. the first byte of the hash (the validation byte) is blank
Next, it reads the signature from the hash and assigns it to the _signature field. It then casts that value to an System.Int32 value, assigns it to the _iterations field, and checks to make sure it is a valid value. If not, it throws a non-descriptive error. Finally, it reads the salt value from the hash and assigns it to the _salt field. If we were to call ComputeHash(...) on an instance of this class that has used this constructor, it will use the iterations and salt value used to create the stored hash to create a new hash. Comparing the newly created password hash value against the stored password hash value for equality would allow us to determine if the password input is the correct password.

Here is the full example:

namespace System.Security.Cryptography
{
    public sealed class PBKDF2Managed : IDisposable
    {
        const int SIGNATURE_SIZE = 4;
        const int SALT_SIZE = 128;
        const int KEY_SIZE = 256;
        const int HASH_SIZE = 1 + SIGNATURE_SIZE + SALT_SIZE + KEY_SIZE;

        int _iterations;
        byte[] _signature;
        byte[] _salt;
        byte[] _key;


        public PBKDF2Managed(int iterations)
        {
            if (iterations < 1)
            {
                throw new CryptographicException("Invalid iteration value.");
            }
            _iterations = iterations;
            _signature = BitConverter.GetBytes(_iterations);
            _salt = new byte[SALT_SIZE];
            using (RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider())
            {
                _rng.GetBytes(_salt);
            }
        }

        public PBKDF2Managed(byte[] salt, int iterations)
        {
            if (iterations < 1)
            {
                throw new CryptographicException("Invalid iteration value.");
            }
            if (salt == null)
            {
                throw new CryptographicException("Invalid salt value.");
            }
            if (salt.Length != SALT_SIZE)
            {
                throw new CryptographicException("Invalid salt value.");
            }
            _iterations = iterations;
            _signature = BitConverter.GetBytes(_iterations);
            _salt = salt;
        }

        public PBKDF2Managed(byte[] hash)
        {
            if (hash == null)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            if (hash.Length != HASH_SIZE)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            if (hash[0] != 0x00)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            _signature = new byte[SIGNATURE_SIZE];
            Buffer.BlockCopy(hash, 1, _signature, 0, 4);
            _iterations = BitConverter.ToInt32(_signature, 0);
            if (_iterations < 1)
            {
                throw new CryptographicException("Invalid hash value.");
            }
            _salt = new byte[SALT_SIZE];
            Buffer.BlockCopy(hash, 1 + SIGNATURE_SIZE, _salt, 0, SALT_SIZE);
        }


        public byte[] ComputeHash(byte[] password)
        {
            if (password == null)
            {
                throw new ArgumentNullException("password");
            }
            using (Rfc2898DeriveBytes _pbkdf2 = new Rfc2898DeriveBytes(password, _salt, _iterations))
            {
                _key = _pbkdf2.GetBytes(KEY_SIZE);
            }
            byte[] _hash = new byte[HASH_SIZE];
            Buffer.BlockCopy(_signature, 0, _hash, 1, SIGNATURE_SIZE);
            Buffer.BlockCopy(_salt, 0, _hash, 1 + SIGNATURE_SIZE, SALT_SIZE);
            Buffer.BlockCopy(_key, 0, _hash, 1 + SIGNATURE_SIZE + SALT_SIZE, KEY_SIZE);
            return _hash;
        }

        public void Dispose()
        {
            // clean up any unmanaged resources
        }
    }
}

Here are some things to consider about the design of the example:

While we provided constant values for certain things, such as salt sizes, key sizes, and hash sizes, we have not even provided a default iterations value. This is so the class can be configured on a "per-application" basis. If a developer feels the need to use a significantly higher amount of iterations, they simply supply the required number in the constructor. Conversely, if a developer knows that the system running their application will be a low-grade, less capable system with minimal processing power, they can adjust the amount of iterations accordingly.

Like the reasoning behind accepting byte array input values in ComputeHash(...) calls, the output of ComputeHash() is an array of bytes as well, and for the same reasoning. It is encoding-agnostic. By our design, you are able to encode your hash output in whichever encoding you see fit and there is no dependency formed towards a particular encoding.

On a final note, let's talk about the format in which our final hash values are written before being returned to the caller. In our example, we leave the first byte blank, followed by the salt, then the hashed key value. This order is not written in stone and you can re-organize the order in which they are written to the final result how you see fit. You can add multiple blank validation bytes in various locations, put the salt last, or first. Doing so makes YOUR implementation that much more unique and less likely to be guessed on how it is structured. Just be sure you adjust your implementation accordingly and I wouldn't suggest separating each portion with a blank byte as this could be a dead giveaway on which part is the salt and which is the actual key. You want that salt to be as protected as possible.

Let's wrap things up with a sample console application to test out our class. We'll use UTF8 encoding for text and Base64 encoding for the hash value and 1000 iterations just for an example:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Security.Cryptography;

namespace System
{
    class Program
    {
        static void Main(string[] args)
        {
            string myPassword = "my super secret password";

            byte[] pwdHash;
            using (var pbkdf2 = new PBKDF2Managed(1000))
            {
                pwdHash = pbkdf2.ComputeHash(Encoding.UTF8.GetBytes(myPassword));
            }

            string base64PwdHash = Convert.ToBase64String(pwdHash);

            // let's try guessing the password
            byte[] guessedPwdHash;
            using (var pbkdf2 = new PBKDF2Managed(pwdHash))
            {
                guessedPwdHash = pbkdf2.ComputeHash(Encoding.UTF8.GetBytes("Just guessing"));
            }

            string base64GuessedPwdHash = Convert.ToBase64String(guessedPwdHash);

            // tell us if we guesed right
            Console.WriteLine(string.Equals(base64PwdHash, base64GuessedPwdHash));

            // ok, let's use the proper password now
            byte[] userPwdHash;
            using (var pbkdf2 = new PBKDF2Managed(pwdHash))
            {
                userPwdHash = pbkdf2.ComputeHash(Encoding.UTF8.GetBytes("my super secret password"));
            }

            string base64UserPwdHash = Convert.ToBase64String(userPwdHash);

            // tell us if we entered the correct password
            Console.WriteLine(string.Equals(base64PwdHash, base64UserPwdHash));

            Console.ReadLine();
        }
    }
}

In the above test, we first create a "super secret password" and hash it. Then we try to validate a password we know is wrong and compare the hashed values. The console will tell us false because it's incorrect (or at least it better!). Next we try the proper password and of course, it tells use true because we entered the right password.

Hopefully this article has been both informative and helpful and as always feel free to use the code examples and modify them as you see fit (just be sure you do so in a secure way) and feel free to ask any questions you may have or share tips in the comments below.

Happy coding!

Tuesday, July 30, 2013

Hexadecimal, Base64 and Binary Encoding in .NET

Many times a .NET developer encounters a scenario where they would like to use specialized encoding for data, such as Base-64 encoding, Hexadecimal (Base-16 or "Hex") encoding, or Binary (Base-2) encoding. The .NET framework has some built in ways to make this possible, but these are not treated as true "encodings" and some are very bad for performance. For instance, the .NET framework contains a method that is part of the System.Convert class for converting to and from base-64 but it is not provided in the form of a true "encoding". As for hex encoding and binary (base-2) encodings, this is done through special formatting strings included in a "ToString()" call, which are notoriously poor performance. While the performance hit may not matter for some applications, it undoubtedly will be noticed in larger, resource-intensive applications.

Wouldn't it be nice to have an actual Encoding class to do this for you? Many consider the though but then shy away when it comes time to create an efficient and accurate algorithm to be used by the encoding. Well let me put your mind at ease and save you some tylenol. I'll show you how to do this properly in a reusable set of classes.

I'm not going to spend much time explaining exactly how the algorithms work as it would be very time consuming to do so, but I assure you they work very fast and very accurately. Custom base-64 encoding has notoriously been a daunting task as it is hard for most to develop an algorithm that properly validates base-64 formatting. When developing these algorithms, I studied the .NET frameworks source code to understand the way Microsoft handles these encoding routines internally for best performance and accuracy. It is sad that they didn't implement public Encoding-derived classes but the following code will give you just that.

To get started, we are going to make each of these Encoding classes derive from the abstract System.Text.Encoding base class.

First, let's create the Base64 encoding class as follows:

using System;
using System.Diagnostics;
using System.Globalization;
using System.Security;

namespace System.Text
{
    public sealed class Base64Encoding : Encoding
    {
        const int MAX_CHAR_COUNT = int.MaxValue / 4 * 3 - 2;

        static byte[] char2val = new byte[128]
        {
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 
            0xFF, 0xFF, 0xFF,   62, 0xFF, 0xFF, 0xFF,   63,
            52, 53, 54, 55, 56, 57, 58, 59,   
            60, 61, 0xFF, 0xFF, 0xFF,   64, 0xFF, 0xFF,
            0xFF, 0, 1, 2, 3, 4, 5, 6,    
            7, 8, 9, 10, 11, 12, 13, 14,
            15, 16, 17, 18, 19, 20, 21, 22,   
            23, 24, 25, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 26, 27, 28, 29, 30, 31, 32,   
            33, 34, 35, 36, 37, 38, 39, 40,
            41, 42, 43, 44, 45, 46, 47, 48,   
            49, 50, 51, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
        };

        static string val2char = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";

        static byte[] val2byte = new byte[]
        {
            (byte)'A',(byte)'B',(byte)'C',(byte)'D',(byte)'E',(byte)'F',(byte)'G',(byte)'H',
            (byte)'I',(byte)'J',(byte)'K',(byte)'L',(byte)'M',(byte)'N',(byte)'O',(byte)'P',
            (byte)'Q',(byte)'R',(byte)'S',(byte)'T',(byte)'U',(byte)'V',(byte)'W',(byte)'X',
            (byte)'Y',(byte)'Z',(byte)'a',(byte)'b',(byte)'c',(byte)'d',(byte)'e',(byte)'f',
            (byte)'g',(byte)'h',(byte)'i',(byte)'j',(byte)'k',(byte)'l',(byte)'m',(byte)'n',
            (byte)'o',(byte)'p',(byte)'q',(byte)'r',(byte)'s',(byte)'t',(byte)'u',(byte)'v',
            (byte)'w',(byte)'x',(byte)'y',(byte)'z',(byte)'0',(byte)'1',(byte)'2',(byte)'3',
            (byte)'4',(byte)'5',(byte)'6',(byte)'7',(byte)'8',(byte)'9',(byte)'+',(byte)'/'
        };


        [SecuritySafeCritical]
        unsafe public override int GetByteCount(char[] chars, int index, int count)
        {
            if (chars == null)
            {
                throw new ArgumentNullException("chars");
            }
            if (index < 0)
            {
                throw new ArgumentOutOfRangeException("index", "Value must be non-negative");
            }
            if (index > chars.Length)
            {
                throw new ArgumentOutOfRangeException("index", "Offset exceeds the buffer size..");
            }
            if (count < 0)
            {
                throw new ArgumentOutOfRangeException("count", "Value must benon-negative.");
            }
            if (count > chars.Length - index)
            {
                throw new ArgumentOutOfRangeException("count", "Value exceeds the remaining buffer size.");
            }
            if (count == 0)
            {
                return 0;
            }
            if ((count % 4) != 0)
            {
                throw new FormatException("Invalid Base64 length.");
            }
            fixed (byte* _char2val = char2val)
            {
                fixed (char* _chars = &chars[index])
                {
                    int totalCount = 0;
                    char* pch = _chars;
                    char* pchMax = _chars + count;
                    while (pch < pchMax)
                    {
                        Debug.Assert(pch + 4 <= pchMax, "");
                        char pch0 = pch[0];
                        char pch1 = pch[1];
                        char pch2 = pch[2];
                        char pch3 = pch[3];

                        if ((pch0 | pch1 | pch2 | pch3) >= 128)
                        {
                            throw new FormatException("Invalid Base64 sequence.");
                        }

                        int v1 = _char2val[pch0];
                        int v2 = _char2val[pch1];
                        int v3 = _char2val[pch2];
                        int v4 = _char2val[pch3];

                        if (!IsValidLeadBytes(v1, v2, v3, v4) || !IsValidTailBytes(v3, v4))
                        {
                            throw new FormatException("Invalid Base64 sequence.");
                        }

                        int byteCount = (v4 != 64 ? 3 : (v3 != 64 ? 2 : 1));
                        totalCount += byteCount;
                        pch += 4;
                    }
                    return totalCount;
                }
            }
        }

        [SecuritySafeCritical]
        unsafe public override int GetBytes(char[] chars, int charIndex, int charCount, byte[] bytes, int byteIndex)
        {
            if (chars == null)
            {
                throw new ArgumentNullException("chars");
            }
            if (charIndex < 0)
            {
                throw new ArgumentOutOfRangeException("charIndex", "Value must non-negative.");
            }
            if (charIndex > chars.Length)
            {
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            }
            if (charCount < 0)
            {
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            }
            if (charCount > chars.Length - charIndex)
            {
                throw new ArgumentOutOfRangeException("charCount", "Size exceeds remaining buffer size.");
            }
            if (bytes == null)
            {
                throw new ArgumentNullException("bytes");
            }
            if (byteIndex < 0)
            {
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative");
            }
            if (byteIndex > bytes.Length)
            {
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            }
            if (charCount == 0)
            {
                return 0;
            }
            if ((charCount % 4) != 0)
            {
                throw new FormatException("Invalid Base64 length.");
            }
            fixed (byte* _char2val = char2val)
            {
                fixed (char* _chars = &chars[charIndex])
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        char* pch = _chars;
                        char* pchMax = _chars + charCount;
                        byte* pb = _bytes;
                        byte* pbMax = _bytes + bytes.Length - byteIndex;
                        while (pch < pchMax)
                        {
                            Debug.Assert(pch + 4 <= pchMax, "");
                            char pch0 = pch[0];
                            char pch1 = pch[1];
                            char pch2 = pch[2];
                            char pch3 = pch[3];

                            if ((pch0 | pch1 | pch2 | pch3) >= 128)
                            {
                                throw new FormatException("Invalid Base64 sequence.");
                            }

                            int v1 = _char2val[pch0];
                            int v2 = _char2val[pch1];
                            int v3 = _char2val[pch2];
                            int v4 = _char2val[pch3];

                            if (!IsValidLeadBytes(v1, v2, v3, v4) || !IsValidTailBytes(v3, v4))
                            {
                                throw new FormatException("Invalid Base64 sequence.");
                            }

                            int byteCount = (v4 != 64 ? 3 : (v3 != 64 ? 2 : 1));
                            if (pb + byteCount > pbMax)
                            {
                                throw new ArgumentException("bytes", "Array is too small.");
                            }

                            pb[0] = (byte)((v1 << 2) | ((v2 >> 4) & 0x03));
                            if (byteCount > 1)
                            {
                                pb[1] = (byte)((v2 << 4) | ((v3 >> 2) & 0x0F));
                                if (byteCount > 2)
                                {
                                    pb[2] = (byte)((v3 << 6) | ((v4 >> 0) & 0x3F));
                                }
                            }
                            pb += byteCount;
                            pch += 4;
                        }
                        return (int)(pb - _bytes);
                    }
                }
            }
        }

        [SecuritySafeCritical]
        unsafe public int GetBytes(byte[] chars, int charIndex, int charCount, byte[] bytes, int byteIndex)
        {
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");

            if (charCount < 0)
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            if (charCount > chars.Length - charIndex)
                throw new ArgumentOutOfRangeException("charCount", "Size exceeds remaining buffer space.");

            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");

            if (charCount == 0)
                return 0;
            if ((charCount % 4) != 0)
                throw new FormatException("Invalid Base64 length.");
            fixed (byte* _char2val = char2val)
            {
                fixed (byte* _chars = &chars[charIndex])
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        byte* pch = _chars;
                        byte* pchMax = _chars + charCount;
                        byte* pb = _bytes;
                        byte* pbMax = _bytes + bytes.Length - byteIndex;
                        while (pch < pchMax)
                        {
                            Debug.Assert(pch + 4 <= pchMax, "");
                            byte pch0 = pch[0];
                            byte pch1 = pch[1];
                            byte pch2 = pch[2];
                            byte pch3 = pch[3];
                            if ((pch0 | pch1 | pch2 | pch3) >= 128)
                                throw new FormatException("Invalid Base64 sequence.");

                            int v1 = _char2val[pch0];
                            int v2 = _char2val[pch1];
                            int v3 = _char2val[pch2];
                            int v4 = _char2val[pch3];

                            if (!IsValidLeadBytes(v1, v2, v3, v4) || !IsValidTailBytes(v3, v4))
                                throw new FormatException("Invalid Base64 sequence.");

                            int byteCount = (v4 != 64 ? 3 : (v3 != 64 ? 2 : 1));
                            if (pb + byteCount > pbMax)
                                throw new ArgumentException("bytes", "Array size too small.");

                            pb[0] = (byte)((v1 << 2) | ((v2 >> 4) & 0x03));
                            if (byteCount > 1)
                            {
                                pb[1] = (byte)((v2 << 4) | ((v3 >> 2) & 0x0F));
                                if (byteCount > 2)
                                {
                                    pb[2] = (byte)((v3 << 6) | ((v4 >> 0) & 0x3F));
                                }
                            }
                            pb += byteCount;
                            pch += 4;
                        }
                        return (int)(pb - _bytes);
                    }
                }
            }
        }

        public override int GetCharCount(byte[] bytes, int index, int count)
        {
            return GetMaxCharCount(count);
        }

        [SecuritySafeCritical]
        unsafe public int GetChars(byte[] bytes, int byteIndex, int byteCount, byte[] chars, int charIndex)
        {
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            if (byteCount < 0)
                throw new ArgumentOutOfRangeException("byteCount", "Value must be non-negative.");
            if (byteCount > bytes.Length - byteIndex)
                throw new ArgumentOutOfRangeException("byteCount", "Exceeds remaining buffer size.");
 
            int charCount = GetCharCount(bytes, byteIndex, byteCount);
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
  
            if (charCount < 0 || charCount > chars.Length - charIndex)
                throw new ArgumentException("chars", "Array is to small.");
 
            // We've computed exactly how many chars there are and verified that
            // there's enough space in the char buffer, so we can proceed without
            // checking the charCount.
 
            if (byteCount > 0)
            {
                fixed (byte* _val2byte = val2byte)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (byte* _chars = &chars[charIndex])
                        {
                            byte* pb = _bytes;
                            byte* pbMax = pb + byteCount - 3;
                            byte* pch = _chars;
 
                            // Convert chunks of 3 bytes to 4 chars
                            while (pb <= pbMax)
                            {
                                pch[0] = _val2byte[(pb[0] >> 2)];
                                pch[1] = _val2byte[((pb[0] & 0x03) << 4) | (pb[1] >> 4)];
                                pch[2] = _val2byte[((pb[1] & 0x0F) << 2) | (pb[2] >> 6)];
                                pch[3] = _val2byte[pb[2] & 0x3F];
 
                                pb += 3;
                                pch += 4;
                            }
  
                            // Handle 1 or 2 trailing bytes
                            if (pb - pbMax == 2)
                            {
                                // 1 trailing byte
                                pch[0] = _val2byte[(pb[0] >> 2)];
                                pch[1] = _val2byte[((pb[0] & 0x03) << 4)];
                                pch[2] = (byte)'=';
                                pch[3] = (byte)'=';
                            }
                            else if (pb - pbMax == 1)
                            {
                                // 2 trailing bytes
                                pch[0] = _val2byte[(pb[0] >> 2)];
                                pch[1] = _val2byte[((pb[0] & 0x03) << 4) | (pb[1] >> 4)];
                                pch[2] = _val2byte[((pb[1] & 0x0F) << 2)];
                                pch[3] = (byte)'=';
                            }
                            else
                            {
                                // 0 trailing bytes
                                Debug.Assert(pb - pbMax == 3, "");
                            }
                        }
                    }
                }
            }
 
            return charCount;
        }

        [SecuritySafeCritical]
        unsafe public override int GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex)
        {
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            if (byteCount < 0)
                throw new ArgumentOutOfRangeException("byteCount", "Value must be non-negative.");
            if (byteCount > bytes.Length - byteIndex)
                throw new ArgumentOutOfRangeException("byteCount", "Size exceeds remaining buffer size.");

            int charCount = GetCharCount(bytes, byteIndex, byteCount);
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            if (charCount < 0 || charCount > chars.Length - charIndex)
                throw new ArgumentException("chars", "Array is too small.");

            // We've computed exactly how many chars there are and verified that
            // there's enough space in the char buffer, so we can proceed without
            // checking the charCount.

            if (byteCount > 0)
            {
                fixed (char* _val2char = val2char)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (char* _chars = &chars[charIndex])
                        {
                            byte* pb = _bytes;
                            byte* pbMax = pb + byteCount - 3;
                            char* pch = _chars;

                            // Convert chunks of 3 bytes to 4 chars
                            while (pb <= pbMax)
                            {
                                pch[0] = _val2char[(pb[0] >> 2)];
                                pch[1] = _val2char[((pb[0] & 0x03) << 4) | (pb[1] >> 4)];
                                pch[2] = _val2char[((pb[1] & 0x0F) << 2) | (pb[2] >> 6)];
                                pch[3] = _val2char[pb[2] & 0x3F];

                                pb += 3;
                                pch += 4;
                            }

                            // Handle 1 or 2 trailing bytes
                            if (pb - pbMax == 2)
                            {
                                // 1 trailing byte
                                pch[0] = _val2char[(pb[0] >> 2)];
                                pch[1] = _val2char[((pb[0] & 0x03) << 4)];
                                pch[2] = '=';
                                pch[3] = '=';
                            }
                            else if (pb - pbMax == 1)
                            {
                                // 2 trailing bytes
                                pch[0] = _val2char[(pb[0] >> 2)];
                                pch[1] = _val2char[((pb[0] & 0x03) << 4) | (pb[1] >> 4)];
                                pch[2] = _val2char[((pb[1] & 0x0F) << 2)];
                                pch[3] = '=';
                            }
                            else
                            {
                                // 0 trailing bytes
                                Debug.Assert(pb - pbMax == 3, "");
                            }
                        }
                    }
                }
            }

            return charCount;
        }
        
        public override int GetMaxByteCount(int charCount)
        {
            if (charCount < 0)
            {
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            }
            if ((charCount % 4) != 0)
            {
                throw new FormatException("Invalid Base64 length.");
            }
            return charCount / 4 * 3;
        }

        public override int GetMaxCharCount(int byteCount)
        {
            if (byteCount < 0 || byteCount > MAX_CHAR_COUNT)
            {
                throw new ArgumentOutOfRangeException("byteCount", 
                    string.Format("Value must be within the range of 0 and {0}", MAX_CHAR_COUNT));
            }
            return ((byteCount + 2) / 3) * 4;
        }

        private bool IsValidLeadBytes(int v1, int v2, int v3, int v4)
        {
            return ((v1 | v2) < 64) && ((v3 | v4) != 0xFF);
        }

        private bool IsValidTailBytes(int v3, int v4)
        {
            return !(v3 == 64 && v4 != 64);
        }

    }
}

I apologize for the length but as you can see, there is a lot going on here. The algorithm make's heavy use of bit-shifting and incorporates static look-up tables for the best performance possible. We are also making heavy use of pointers here. This is done to avoid any unnecessary reference calls, reducing the overhead cost and dramatically increasing performance. When we are using our pointers, we are also using the "fixed" keyword to make sure that the references the pointers point to do not get moved in memory while we are using them. This is a very important point. If we didn't "fix" our references in memory, we would have no way of being sure that the runtime hasn't moved our references to another memory address and could result in memory leaks and unexpected results. We override the important methods from the abstract base class that will be used by the "plumbing" to allow us to use calls to GetBytes(...) and GetString(...) calls when using this class. You may have also noticed the use of the [SecuritySafeCritical] attribute. This is used to let the compiler know that although there is code being used in an "unsafe" context (via pointers), that it should not give warnings when used in code that hasn't specified to allow unsafe code.

Next we'll create the Binary encoding class as follows:

using System;
using System.Diagnostics;
using System.Globalization;
using System.Security;

namespace System.Text
{
    public sealed class BinaryEncoding : Encoding
    {
        const int MAX_CHAR_COUNT = int.MaxValue / 8;

        static string val2char = "01";

        public override int GetByteCount(char[] chars, int index, int count)
        {
            return GetMaxByteCount(count);
        }

        [SecuritySafeCritical]
        unsafe public override int GetBytes(char[] chars, int charIndex, int charCount, byte[] bytes, int byteIndex)
        {
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            if (charCount < 0)
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            if (charCount > chars.Length - charIndex)
                throw new ArgumentOutOfRangeException("charCount", "Size exceeds remaining buffer space.");
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            int byteCount = GetByteCount(chars, charIndex, charCount);
            if (byteCount < 0 || byteCount > bytes.Length - byteIndex)
                throw new ArgumentException("bytes", "Array size is too small.");
            if (charCount > 0)
            {
                Array.Reverse(chars);
                fixed (char* _val2char = val2char)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (char* _chars = &chars[charIndex])
                        {
                            char* pch = _chars;
                            char* pchMax = _chars + charCount;
                            byte* pb = _bytes;
                            while (pch < pchMax)
                            {
                                Debug.Assert(pch + 8 <= pchMax, "");
                                if (pch[0] == _val2char[1]) pb[0] |= (1 << 0);
                                if (pch[1] == _val2char[1]) pb[0] |= (1 << 1);
                                if (pch[2] == _val2char[1]) pb[0] |= (1 << 2);
                                if (pch[3] == _val2char[1]) pb[0] |= (1 << 3);
                                if (pch[4] == _val2char[1]) pb[0] |= (1 << 4);
                                if (pch[5] == _val2char[1]) pb[0] |= (1 << 5);
                                if (pch[6] == _val2char[1]) pb[0] |= (1 << 6);
                                if (pch[7] == _val2char[1]) pb[0] |= (1 << 7);
                                pb++;
                                pch += 8;
                            }
                        }
                    }
                }
            }
            return byteCount;
        }

        public override int GetCharCount(byte[] bytes, int index, int count)
        {
            return GetMaxCharCount(count);
        }

        [SecuritySafeCritical]
        unsafe public override int GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex)
        {
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            if (byteCount < 0)
                throw new ArgumentOutOfRangeException("byteCount", "Value must be non-negative.");
            if (byteCount > bytes.Length - byteIndex)
                throw new ArgumentOutOfRangeException("byteCount", "Size exceeds remaining buffer space.");
            int charCount = GetCharCount(bytes, byteIndex, byteCount);
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            if (charCount < 0 || charCount > chars.Length - charIndex)
                throw new ArgumentException("chars", "Array size is too small.");
            if (byteCount > 0)
            {
                Array.Reverse(chars);
                fixed (char* _val2char = val2char)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (char* _chars = &chars[charIndex])
                        {
                            char* pch = _chars;
                            byte* pb = _bytes;
                            byte* pbMax = _bytes + byteCount;
                            while (pb < pbMax)
                            {
                                pch[0] = ((pb[0] & 1) > 0) ? _val2char[1] : _val2char[0];
                                pch[1] = ((pb[0] & 2) > 0) ? _val2char[1] : _val2char[0];
                                pch[2] = ((pb[0] & 4) > 0) ? _val2char[1] : _val2char[0];
                                pch[3] = ((pb[0] & 8) > 0) ? _val2char[1] : _val2char[0];
                                pch[4] = ((pb[0] & 16) > 0) ? _val2char[1] : _val2char[0];
                                pch[5] = ((pb[0] & 32) > 0) ? _val2char[1] : _val2char[0];
                                pch[6] = ((pb[0] & 64) > 0) ? _val2char[1] : _val2char[0];
                                pch[7] = ((pb[0] & 128) > 0) ? _val2char[1] : _val2char[0];
                                pb++;
                                pch += 8;
                            }
                        }
                    }
                }
            }
            Array.Reverse(chars);
            return charCount;
        }

        public override int GetMaxByteCount(int charCount)
        {
            if (charCount < 0)
            {
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            }
            if ((charCount % 8) != 0)
            {
                throw new FormatException("Invalid binary length.");
            }
            return charCount / 8;
        }

        public override int GetMaxCharCount(int byteCount)
        {
            if (byteCount < 0 || byteCount > int.MaxValue / 8)
            {
                throw new ArgumentOutOfRangeException("byteCount", 
                    string.Format("Value must be within range of 0 and {0}.", MAX_CHAR_COUNT));
            }
            return byteCount * 8;
        }
    }
}

A little less going on in the Binary encoding class due to few possible character values (only 1's and 0's).

Finally, lets create the Hexadecimal encoding class as follows:

using System;
using System.Diagnostics;
using System.Globalization;
using System.Security;

namespace System.Text
{
    public sealed class HexEncoding : Encoding
    {
        const int MAX_CHAR_COUNT = int.MaxValue / 2;

        static byte[] char2val = new byte[128]
        {
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
            0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
        };

        static string val2char = "0123456789ABCDEF";


        static HexEncoding()
        {
            for (char ch = '0'; ch <= '9'; ch++)
                Debug.Assert(char2val[ch] == ch - '0', "");

            for (char ch = 'A'; ch <= 'F'; ch++)
                Debug.Assert(char2val[ch] == ch - 'A' + 10, "");

            for (char ch = 'a'; ch <= 'f'; ch++)
                Debug.Assert(char2val[ch] == ch - 'a' + 10, "");
        }


        public override int GetByteCount(char[] chars, int index, int count)
        {
            return GetMaxByteCount(count);
        }

        [SecuritySafeCritical]
        unsafe public override int GetBytes(char[] chars, int charIndex, int charCount, byte[] bytes, int byteIndex)
        {
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            if (charCount < 0)
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            if (charCount > chars.Length - charIndex)
                throw new ArgumentOutOfRangeException("charCount", "Size exceeds remaining buffer space.");
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            int byteCount = GetByteCount(chars, charIndex, charCount);
            if (byteCount < 0 || byteCount > bytes.Length - byteIndex)
                throw new ArgumentException("bytes", "Array size is too small.");
            if (charCount > 0)
            {
                fixed (byte* _char2val = char2val)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (char* _chars = &chars[charIndex])
                        {
                            char* pch = _chars;
                            char* pchMax = _chars + charCount;
                            byte* pb = _bytes;
                            while (pch < pchMax)
                            {
                                Debug.Assert(pch + 2 <= pchMax, "");
                                char pch0 = pch[0];
                                char pch1 = pch[1];
                                if ((pch0 | pch1) >= 128)
                                    throw new FormatException("Invalid hexadecimal sequence.");
                                byte d1 = _char2val[pch0];
                                byte d2 = _char2val[pch1];
                                if ((d1 | d2) == 0xFF)
                                    throw new FormatException("Invalid hexadecimal sequence.");
                                pb[0] = (byte)((d1 << 4) + d2);
                                pch += 2;
                                pb++;
                            }
                        }
                    }
                }
            }
            return byteCount;
        }

        public override int GetCharCount(byte[] bytes, int index, int count)
        {
            return GetMaxCharCount(count);
        }

        [SecuritySafeCritical]
        unsafe public override int GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex)
        {
            if (bytes == null)
                throw new ArgumentNullException("bytes");
            if (byteIndex < 0)
                throw new ArgumentOutOfRangeException("byteIndex", "Value must be non-negative.");
            if (byteIndex > bytes.Length)
                throw new ArgumentOutOfRangeException("byteIndex", "Offset exceeds buffer size.");
            if (byteCount < 0)
                throw new ArgumentOutOfRangeException("byteCount", "Value must be non-negative.");
            if (byteCount > bytes.Length - byteIndex)
                throw new ArgumentOutOfRangeException("byteCount", "Size exceeds remaining buffer space.");
            int charCount = GetCharCount(bytes, byteIndex, byteCount);
            if (chars == null)
                throw new ArgumentNullException("chars");
            if (charIndex < 0)
                throw new ArgumentOutOfRangeException("charIndex", "Value must be non-negative.");
            if (charIndex > chars.Length)
                throw new ArgumentOutOfRangeException("charIndex", "Offset exceeds buffer size.");
            if (charCount < 0 || charCount > chars.Length - charIndex)
                throw new ArgumentException("chars", "Array size is too small.");
            if (byteCount > 0)
            {
                fixed (char* _val2char = val2char)
                {
                    fixed (byte* _bytes = &bytes[byteIndex])
                    {
                        fixed (char* _chars = &chars[charIndex])
                        {
                            char* pch = _chars;
                            byte* pb = _bytes;
                            byte* pbMax = _bytes + byteCount;
                            while (pb < pbMax)
                            {
                                pch[0] = _val2char[pb[0] >> 4];
                                pch[1] = _val2char[pb[0] & 0x0F];
                                pb++;
                                pch += 2;
                            }
                        }
                    }
                }
            }
            return charCount;
        }

        public override int GetMaxByteCount(int charCount)
        {
            if (charCount < 0)
            {
                throw new ArgumentOutOfRangeException("charCount", "Value must be non-negative.");
            }
            if ((charCount % 2) != 0)
            {
                throw new FormatException("Invalid hexadecimal length.");
            }
            return charCount / 2;
        }

        public override int GetMaxCharCount(int byteCount)
        {
            if (byteCount < 0 || byteCount > int.MaxValue / 2)
            {
                throw new ArgumentOutOfRangeException("byteCount", 
                    string.Format("Value must be within range of 0 and {0}.", MAX_CHAR_COUNT));
            }
            return byteCount * 2;
        }
    }
}

There you have it! These classes will ensure valid encoding and decoding of their respective types and are used just like any other encoding class such as UTF8, ASCII, etc.

Unfortunately, we are not able to attach these classes as static properties to the Encoding class that would have allowed us to call them like: Encoding.Base64.GetBytes() or Encoding.Binary.GetString() because Encoding is an abstract class and extension methods may only be attached to object instances.

Something we can do however, is create our own class to provide easy access to encoding methods. Let's do that just for demonstration purposes. We're going to make this class provide static access to all expected encoding types the abstract Encoding class provides, plus we'll include our own as well.

The class will be called Encodings. Create the Encodings class as follows:

namespace System.Text
{
    public sealed class Encodings
    {
        private static readonly Encoding binaryEncoding = new BinaryEncoding();
        private static readonly Encoding base64Encoding = new Base64Encoding();
        private static readonly Encoding hexEncoding = new HexEncoding();

        public static Encoding ASCII
        {
            get { return Encoding.ASCII; }
        }

        public static Encoding Hex
        {
            get { return hexEncoding; }
        }

        public static Encoding Binary
        {
            get { return binaryEncoding; }
        }

        public static Encoding Base64
        {
            get { return base64Encoding; }
        }

        public static Encoding BigEndianUnicode
        {
            get { return Encoding.BigEndianUnicode; }
        }

        public static Encoding Default
        {
            get { return Encoding.Default; }
        }

        public static Encoding Unicode
        {
            get { return Encoding.Unicode; }
        }

        public static Encoding UTF32
        {
            get { return Encoding.UTF32; }
        }

        public static Encoding UTF7
        {
            get { return Encoding.UTF7; }
        }

        public static Encoding UTF8
        {
            get { return Encoding.UTF8; }
        }
    }
}

Unlike the System.Text.Encoding class, this class contains no methods itself. It is simply designed as a way to access static instances of each encoding type. I marked the class as sealed incase you choose to add other functionality to the class, but if you don't plan on doing so, you could mark this class as static, since it contains only static members.

Let's give our awesome new encodings a test run! Create a console application that either has these classes in it or references the assembly that contains them (if created in a class library, which I would personally suggest).

In the console applications Main() method in the Program.cs class, add a using reference to the System.Text namespace, since this where our classes reside. Next, modify your Main() method to match this:

        static void Main(string[] args)
        {
            string s = "Hello world!";

            var binaryEncoded = Encodings.Binary.GetString(Encodings.UTF8.GetBytes(s));

            var binaryDecoded = Encodings.UTF8.GetString(Encodings.Binary.GetBytes(binaryEncoded));

            var hexEncoded = Encodings.Hex.GetString(Encodings.UTF8.GetBytes(s));

            var hexDecoded = Encodings.UTF8.GetString(Encodings.Hex.GetBytes(hexEncoded));

            var base64Encoded = Encodings.Base64.GetString(Encodings.UTF8.GetBytes(s));

            var base64Decoded = Encodings.UTF8.GetString(Encodings.Base64.GetBytes(base64Encoded));


            Console.WriteLine(s);

            Console.WriteLine(binaryEncoded);

            Console.WriteLine(binaryDecoded);

            Console.WriteLine(hexEncoded);

            Console.WriteLine(hexDecoded);

            Console.WriteLine(base64Encoded);

            Console.WriteLine(base64Decoded);


            Console.ReadLine();
        }

If you get curious and want to benchmark the above classes, you will find they are right on par with the native encoders in the .NET framework. Congratulations! You just did something many programmers have shuttered at the thought of, and you barely even broke a sweat doing it!

As always, feel free to use the code provided in this tutorial or modify it anyway you see fit as I am providing it completely open-source, (of course a mention of gratitude would always be appreciated :) ). Feel free to post questions or any cool modifications you come up with in the comments below. Happy coding!

Saturday, April 6, 2013

Including Lua 5.x in Visual Studio 2012 for Embedding in C++

Lua is an incredibly powerful dynamically-typed scripting language. Seriously. Anyone that thinks Perl is the king of scripting has clearly not spent some intimate time with Lua. It can process literally millions of large rows in less than a second. It is this speed and power that makes it a very desirable tool to embed in high-stress systems. Many get their first taste of Lua from video games, a great example being Blizzard Entertainment's epic MMORPG World of Warcraft. Recently, even the powerful online information giant Wikipedia made the leap to implement Lua (kudos to them, I might add!).

It's really a quite sad story how Javascript became the scripting language of choice for the web, rather than Lua. A clever play on the name of a completely unrelated technology at it's emergence (Java) left Lua out in the cold. A sad loss for the web really, but that's a whole other story.

The point is, you don't have to let your applications miss out! Embedding Lua in C++ is a great choice for many things. Sadly, finding a straight-forward answer on how to do so can be a rather discouraging task. Especially if you are working with Visual c++ in Visual Studio. The great news is, it's actually quite easy to do and I'm here to show you exactly how to do it!

The first thing you will need is a suitable Lua build. The current version is 5.2.x. The following approach I'm going to show you is interchangeable with both Lua 5.1 and Lua 5.2. While I will be demonstrating how to do this in Visual Studio 2012, this will also work in Visual Studio 2010.

There are 2 routes you can go about this. You can download the source from Lua.Org directly, or you can get a windows-centric distribution from Lua for Windows, which installs via a simple msi installer and includes a whole plethora of helpful extension libraries for almost anything you could think of doing with Lua. The catch with LuaForWindows is, it's Lua 5.1 and will not be carried on to future versions of Lua. If you get the source from Lua.Org you will have run a build on it yourself (I may be including an article on doing this soon, as it's very easy to do).

Once you have your build of Lua, you are ready to embed it in your c++ project. To do this, you need to tell Visual Studio's compiler and linker where to look for it and what libraries to include.

Let's start with telling it where to look.

I'm going to assume you have installed Lua to the default "c:\Program Files (x86)" directory on your machine. If you have not, don't worry, we'll cover that too.

With your project in visual studio, open the project's "Properties" window. Go to the "VC++ Directories" tab. In here, you will need to edit "Include Directories", "Reference Directories" and "Library Directories". To edit these values, you click on the value, then click on the arrow box that will appear in the top-right side of the value box, and select edit. The window that pops up will allow you to edit the values for the particular line.

First set the "Include Directories" value. If you have installed Lua to the ProgramFiles (x86) directory as suggested above, you can use the following simple macro to point to the directory:

$(ProgramFiles)\Lua\5.1\include

If your directory looks a bit different, just adjust the above text accordingly (example: \Lua\lua-5.1\include).

If you have Lua installed to another location, you can simply use the file navigation tool that appears in the editor window to navigate to the location and point to it. The important part is, you want it pointing to the "/include" folder within the Lua directory.

The process for the remaining settings are the same procedure. Below is a list of each setting and the value to set it to. Remember to adjust the file path if needed.


Include Directories: $(ProgramFiles)\Lua\5.1\include as demonstrated above.
Reference Directories: $(ProgramFiles)\Lua\5.1\lib points to the "lib" folder
Library Directories: $(ProgramFiles)\Lua\5.1\lib points to the "lib" folder

Now visual studio knows what directories to look in when searching for file dependencies. The last part to do is to tell it what lib(s) specifically to include.

In the properties window, go to the "Linker" tab, select the "Input" tab within that, and we will be editing the "Additional Dependencies" value.

As before, click the line, then click the arrow box and select edit. This time we simply need to tell it what lib to use. The only important thing to note is that any libs we specify in here need to be on separate lines.

Enter the following value:

lua5.1.lib

If it is a different version you are using, just adjust the value to the appropriate version.

That's it! Lua is now successfully referenced in our application and ready to be implemented. To see for yourself, try including the c++-specific Lua header file in code:

#include <lua.hpp>

You will notice at the bottom of the visual studio window it is loading the dependencies. If it is giving you error messages that things can not be found, you will need to double check that you have referenced the correct directory locations.

Congratulations! You are ready to use Lua in your application! I will be doing an article about setting up the Lua instance within your app but if you MUST get some info on how to do it, Check out the online version of the book Programming in Lua here. It is written by one of the founders of Lua and includes everything you need to get started.

Cryptographically Secure Pseudo-Random Number Generation in .NET

Many times we need to generate random variables for numerous reasons. When a trivial value is required, the Random class can be just fine but when you need a value that will be used to keep something unique and secure, the Random class simply will not do. This is where the RNGCryptoServiceProvider class comes in to play.

Most would be quick to simply pump out whatever value they need from this class and assume all is well, but beware, as this may not be the case.

If the value will be used to keep something secure, it needs to be cryptographically secure and while that is the purpose of the RNGCryptoServiceProvider, it is still up to you to make sure that the value is even capable of being secure.

Entropy is a term used to describe the randomness of a value. According to current guidelines, a value must contain at least 64 bits of entropy to be considered cryptographically secure. That means, it must contain 64 truly random bits, at a minimum. The reasoning of this is that with current computing power, anything less that 64 random bits would be a relatively trivial task for a machine to guess by sequential generation of values. I'm not saying this can't be done easily with a very powerful system or, as many "hackers" are doing these days, pushing the work on to a very powerful GPU, but we should be safe to assume that the entry point for secure random values will start at 64 bit of entropy.

Lets take a look at how this can be done in a re-usable class, by creating an example. We'll call it CSPRNG (Cryptographically Secure Pseudo Random Number Generator).

    public static class CSPRNG
    {
        public static byte[] GetBytes(int entropy)
        {
            if (entropy < 8)
            {
                throw new ArgumentOutOfRangeException("entropy",
                    "Entropy must be 64 bits or greater to be cryptographically secure.");
            }
            using (RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider())
            {
                byte[] _obj = new byte[entropy];
                _rng.GetBytes(_obj);
                return _obj;
            }
        }

        public static Guid GetGuid()
        {
            using (RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider())
            {
                byte[] _obj = new byte[16];
                _rng.GetBytes(_obj);
                return new Guid(_obj);
            }
        }

        public static long GetInt64(bool allowNegativeValue = false)
        {
            using (RNGCryptoServiceProvider _rng = new RNGCryptoServiceProvider())
            {
                byte[] _obj = new byte[8];
                _rng.GetBytes(_obj);
                long _num = BitConverter.ToInt64(_obj, 0);
                if (!allowNegativeValue)
                {
                    _num = (_num & ~(1L << 63));
                }
                return _num;
            }
        }
    }

This static class is capable of creating 3 different types of variables for use: a byte array with a minimum length of 8 bytes(64 bits), a Guid(128 bits) and an Int64 value(64 bits).

You will notice that the above class does NOT allow any value less than 64 bits (or 8 bytes) to be generated. If it is requested to do so, it will throw an error. This is so we can be assured that this class will ONLY create cryptographically secure values.

The Int64 generator features an optional parameter that controls if negative values are allowed or not. It does so by guaranteeing the most significant bit is ALWAYS turned-off when non-negative values are not allowed. Technically, this could be an issue. A signed Int64 value uses the most significant bit to indicate the signage of the value (positive or negative). By forcing positive values, we are making this value only 63 bits of entropy, but I allow this 1 exception to break the rule for the simple reason that sometimes a negative value will break the requirements of the caller. Another option would be to create an UInt64 (unsigned 64 bit integer) to ensure that all 64 bits are used for entropy.

Some may ask why a Guid would be included, as we could just simply create a seemingly random Guid from the classes static NewGuid() constructor. The reason is this: the static Guid constructor guarantees uniqueness but it does not guarantee a cryptographically secure value because it is not truly random. I've seen many posts on the web about how it would take trillions of years to create a collision and I assure you, they need to check their math, as that is not the case. On a sub-standard system, a duplicate value could be created in ~200 years. Yes that's a lot, but put that same algorithm on to a machine designed for brute force attacks and that number drops drastically.

Hope this sheds some light on keeping random values secure and feel free to use or modify the code above as you see fit.

Thursday, September 20, 2012

The ASP.NET 4.5 MachineKey class brings serious security to the table!

The MachineKey class of .NET 4.0 was a real breath of fresh air. It simplified the task of cryptographically protecting sensitive data into just a few short lines of code. It could be effortlessly configured to function in web farms and cloud-based solutions by simply synchronizing the machine keys across all servers. You could feed it practically anything and it never complained. It's level of security was excellent and was easily customized to suit pretty much whatever needs you had.

Encrypting was as simple as

string encryptedString = MachineKey.Encode(Encoding.UTF8.GetBytes(mySecretString),                    
                                        MachineKeyProtection.All);


and decrypting was as simple as

byte[] decryptedData = MachineKey.Decode(encryptedString, MachineKeyProtection.All);


This could be easily fitted into custom methods to provide numerous helpful tasks, such as generating salted password encryptions to be stored in databases, encrypting serialized objects to be stored in cookies (such as custom forms authentication-type implementations or sessions), protecting shopping cart data. The list goes on.

The MachineKeyProtection enumeration parameter allows you to specify exactly what level of protection you wish to implement, which can be helpful for specific needs. Say you need to ensure something is heavily protected and would like to include an HMAC for validation. The "All" option does this for you. Maybe you have a piece of data that you would like to protect, but it's security isn't mission critical and performance speed is more the priority (as in, say, a large collection of small objects). The "Encrypt" parameter would most likely be the enumeration of choice. Point is, it allows for tailoring to specific scenarios.

Enter the .NET 4.5 revamp!

At first glance, I was a little unsure. While the revamped 4.5 version of the MachineKey class contains the now depreciated Encode and Decode methods, it now has two new methods:

MachineKey.Protect(byte[] data, params string[] purposes)


and

MachineKey.Unprotect(byte[] protectedData, params string[] purposes)


My first thought was "ughh... what did you do to my MachineKey?!?!". After checking up on documentation that "ughh" feeling turned into a "WOW!".

The idea behind the "purposes" parameter(s) is it provides a way to supply, as the parameter name suggests, purposes for the data being protected. Think of it as being able to supply contextual boundaries for the use of the data. You'll notice that this parameter is also decorated with the params modifier, allowing for a variable amount of string parameters to be supplied, such as:

byte[] myUnprotectedBytes = Encoding.UTF8.GetBytes("Hello World!");
byte[] myProtectedBytes = MachineKey.Protect(myUnprotectedBytes, "some special pupose");



Let's imagine that the purpose "some special purpose" really is some special reason that the string "Hello World!" needs to be protected from prying eyes.
Here's the cool part: Since I supplied "some special purpose" as a purpose parameter during the Protect() call, for the data to be unprotected again and restored to it's previous state, the exact same case-sensitive purpose(s) MUST be supplied during the Unprotect() call:

byte[] myUnprotectedBytes = MachineKey.Unprotect(myProtectedBytes, "some special pupose");
string originalString = Encoding.UTF8.GetString(myUnprotectedBytes);


Like I said earlier, the params string[] purposes parameters provide a sort of contextual boundary for the data. If I had instead supplied a different string such as "GIVE ME THE SECRET!" as a purpose during the Unprotect() call, I would have received a rather abstract exception thrown stating "a cryptographic error occurred", which is also nice, as it doesn't give too much details about what exactly went wrong, but we know what happened and that's a nice feature in my opinion as well. This would have also occurred if I had provided the proper purpose and included additional string parameters that were not used during the Protect() call.
Now, I don't know for sure, so don't quote me, but I think the methods may be internally using the purposes params as a form of "salting" the encryption as its being processed. Again, this is only speculation so don't quote me. It just seems this way to me.

You may have noticed that with the new 4.5 methods, we no longer are required to pass a MachineKeyProtection enumeration as a parameter. This is because the new approach is to now apply all protection by default, which should be viewed as a big plus. I will say however, that although the old methods are now depreciated, there may still be occasions where you may choose to use the old methods for the sake of specifying to only "Encrypt" without an HMAC and without validation of purposes for the data in less security-sensitive, more performance-sensitive scenarios.

You may have also noticed that the new methods do not directly take strings as input data or return strings as return values. They deal entirely with bytes. Don't view this as having more work to do. View this as the framework relinquishing FULL control of our data to us, the programmers! Essentially it modularized the specific processes of protecting/unprotecting data, and allows us to decide how we want to handle our data before and after the calls to the API.

If the full weight of this incredible revamp to include purposes is not yet apparent to you, fret not for I will shed some light and get those creative wheels turning for you!

Obviously, purposes provided an enormous security boost. It's like requiring 1, or even many passwords to access the data. But there's much more to this than meets the eye...

These purpose parameters can have as little, or as much meaning as you choose. They can be dynamic, user-specific values, application-specific values, session-specific values. You name it. A programmer I respect very much, Brock Allen covered a great scenario used in a cookie-based temp data API using a very clever implementation of dynamically using the Identity property of the current thread principal as a purpose when protecting and unprotecting, ensuring that only the intended user may access the data. You can view his post here.

You could even take this a step further and create a custom class that is designed to act as a "Purpose Factory" to dynamically handle purposes for specific scenarios or you could even create an entire hierarchy of classes or enums designed to provided more specific purposes for different scenarios.

Anyways, simply put, the new MachineKey of .NET 4.5 is a game changer and I suspect we'll be seeing some pretty clever ideas coming out of the woodworks with it.

My new Blog is here!

I've been meaning to get around to setting up a blog for awhile now where I could share code ideas and samples. Finally, I've decided to get it started. It feels a little strange because I'm more of a framework, server-side kind of guy so a blog, as funny as it sounds, feels very foreign to me. I'm sure I'll get the hang of it soon enough though and, of course, feel free to make suggestions or requests as you see fit.

Much like my diverse span of personal interests, I hope this blog will grow to cover a multitude of topics relating to all that is code! I hope that the topics discussed will be of use to people and inspire you to be creative and think outside of the box. Creativity is the mother of innovation!

I'd also like to take a quick second to say that while I welcome comments, questions and suggestions (including corrections where required, we all make mistakes sometimes) and will always do my best to respond to relevant posts, spam will not be tolerated. Please be kind and considerate to others who may be reading posts here.

With that said, thank you for dropping by and I look forward to hearing from you!