Recently, since Fading importance and the utility of lists, I've been thinking about the top-level approach/process that I use for doing development work, generally.

The main reason was that I was wondering how different other disciplines of development might approach doing work and how much commonality there might be in what I generally do. So I thought it would be interesting to outline my conceptual flow when approaching new work.

A typical strategy that I take to delivering software within the last couple of years, which has focused on being more agile, i.e, exposing and sharing issues more readily and making the strategy design process more collaborative, looks something like this:

 See Strategy for Designing Software Solutions for an illustration of the process

 Sharing the responsibility with your group

The stages of this process initially are from the developer's (my) perspective who has been tasked with delivering the software solution. This means that at each step the developer is cognitively involved in rationalising the information and tasks at each step to develop a mental blueprint of the upcoming work that he/she needs to deliver. This means that much of the process is internalised by the individual developer, and any artefacts that are externalised/produced are from the developer such as diagrams, documentation etc is initially produced to aid their own understanding and solidify their development methodology.

Personal artefacts are the outcomes of the process carried out by the developer and can include designs/diagrams, documentation and code

While the process is initially a personal approach initiated and implemented by the developer, many outcomes can be exposed to the group, particularly unknowns and concerns that the developer might have picked up while defining their process.

A problem with personal exposure to the process is that the amount of responsibility and burden that is implicitly on the developer grows as the process is followed and more information is uncovered. This is especially true of discovered unknowns that need to be accounted for or potential concerns or shortcomings that the developer realises not only within the solutions but within their own personal skillset for example.

Holding onto these concerns, unknowns and problems can be psychologically taxing as they weigh down on the developer during subsequent stages of the process such as during Technical solution implementation for example. This is where group exposure to these issues can serve as a relief to these burdens, as it can distribute and share the responsibility of dealing with them, including sourcing potential solutions. This also allows for expectations to be readjusted as many issues are likely to impact the Technical solution implementation stage which is of principal concern for the individual developer as well as the group. This is likely to decrease the pressure that the developer faces when having to deliver the technical solution, as it involves more people (team) and distributes some of the burden such that technical solution implementation can proceed with more clarity with the impact that these issues will have on the technical implementation.

In this way, there needs to be a shared notion of the process where the personal process extends into the group and vice versa, however much of the process is initially heavily centred and focused around the engineer's personal ability to define it. It then needs to be fed back into the group to share and initiate feedback that serves both the developer and the group depending on the developer to deliver the software solution.

Group artefacts are resources that allow for exposure and feedback back to the developer's process and can meetings, design/documentation review, project/goal/strategy/planning tracking and code review.

Discussion

Generally, I find it useful to produce an artefact that represents my personal outcomes or thinking, such that they can be shared with the team, both for feedback but also for relieving the pressure from dealing with unknowns/uncertainties/issues etc. This includes taking a strategy that keeps track of my progress as outlined in Fading importance and the utility of lists which can help to quantify your efforts throughout the above process.

I would say a large portion of the process is similar and has parity with the typical software development process where requirement gathering proceeds analysis and which leads to design and then ultimately implementation. However, my process presents this slightly differently as more of a personal development process that is followed by an individual contributor (developer) of the overall software development lifecycle of a product or feature.

Also, in the process, there is an emphasis on the initiative that the developer must have when encountering issues, as the developer is often the first person to become aware of them, and failure to expediently expose issues can result in unnecessary pressure in dealing with them. So in this way, this process factors in these personal concerns. 

The process tracks individual concerns and processes such as research and the need for understanding and idea modelling. It implies that requirements are already gathered and just need to be correctly understood, or if they are poorly gathered, then that it's the developer's responsibility to determine what the requirements are and then understand them in order to develop a solution to deliver it.

I think much of the business analyst role especially for more agile, smaller teams are now largely deprecated and much of this responsibility is falling on the individual experienced developers.

There is also generally the feeling in my mind that more responsibility is required of developers to deliver end-to-end solutions, and I suspect this is because the complexity of software is difficult to adequately distribute across multiple actors such as an analyst, project manager and other developers.

So more of these tasks are taken on by the individual developer and this requires more skills and better communication skills (which arguably the other roles mentioned previously would be more traditionally good at). However, it's arguably easier to distribute the development tasks to other developers, provided the tasks have been defined - which again is usually the work of another developer if indeed the tasks get distributed at all. In some cases, the developer is responsible for the entire end-to-end software solution due to the difficulty in communicating the complexity of the implementation to others in an adequate time, and so the work takes as long as it takes the developer who understands it to implement. 

I think there is a concern in software projects, that due to the complexity of software projects, both theoretically but more problematically at a technical level, it is becoming difficult to speed up development by bringing in new developers as the time to transfer understanding of the complexity is now longer than the time required to deliver the entire solution. So in some cases, solution delivery is not being given strict deadlines but they are progressively being delivered piecemeal as they are being incrementally implemented by the developers who understand the solution/problem/complexity.

In the process outlined, understanding requirements, designing ideas and thinking about how to model and implement them, theoretically and technically is left to the individual developer to satisfy, requiring and drawing knowledge from past experience and training. This suggests that the quality of the developer is being more crucially being recognised as important for their ability to understand, contextualize, rationalize simplify and execute the delivery of solutions in light of the increasing complexities of software development generally.

An interesting discussion that is not touched upon here is how this process differs from the typical game development process, and how either it or this process can lean from each other. For example, is there a similarity to requiring research as part of a task or communicating unknowns to the team, and how is this carried out? I imagine there would be some similarity to non-game development but to what extent or how does it differ?

Since Christmas Period Tinkering, I've been attending C++ classes and we've been getting various tasks to do. Last week we were told to take a normal string and print some stats about it. For example, we needed to count the number of words in the string, the occurrences of characters in the string and count how many upper and lower case characters there were. We could only use basic C arrays, and you cannot modify the string. No STL.

I thought this was a trivial problem, having previously achieved this with C# with ease through using library functions like String.Split(), String.Trim() etc. so I thought this would be relatively trivial. 

I was surprised at how long it took me to count the number of words in a string! There are string variances that need to be accommodated, i.e exceptions. For example, the string "The quick brown fox jumps over the lazy dog" has 9 words, and if you count the number of spaces, which is arguably the first thing most people would do, you soon encounter problems with variations of this string such as when there are leading or trailing spaces. Once this is solved, extra spaces in between words cause problems. Soon it becomes clear that counting spaces are not worth it. 

One way around this problem is to iterate through each character in the string and determine if the character is part of a word or not. It pretty much solves the leading and trailing issue and the original string is not modified:

#include <cmath>
#include <iostream>

using std :: cout;
using std :: cin;
using std :: endl;

int main(int argc, char* argv[])
{
	char myText[] = "The quick brown fox jumps over the lazy dog";
	cout << myText << endl;
	
	// 1. count characters not incl null
	int count_items = sizeof(myText) / sizeof(myText[0]);	
	int count_chars = count_items - 1;

	int count_upper = 0;
	int count_lower = 0;		
	int word_count = 0;
	bool in_word = false;
	
	const int MAX_CHARACTERS = 128; 
	int characters_counts[MAX_CHARACTERS] = {0};

	for(int c = 0; c <= count_chars-1; c++)
	{
		// Not in a word
		if(isspace(myText[c])) 
		{			
			in_word = false;
		}
		else
		{				
			 // if was not previously in a word, then we are in a new word, increase word count
			if(!in_word)
			{				
				word_count++;
			}			
			in_word = true; // In a word
		}

		// 2. identify if character is upper csase or lowercase (functions)
		
		if(isupper(myText[c]))
		{			
			count_upper++;		
		}		

		if(islower(myText[c]))
		{		
			count_lower++;
		}	

		// 3. count number given characters eg space in the string []
		characters_counts[myText[c]]++;
	}

	cout << endl;
	cout << "Total character count: " << count_chars << endl;
	cout << "Uppercase count: " << count_upper << endl;
	cout << "Lowercase count: " << count_lower << endl;

	// 4. count the number of words in the string
	cout << "Word count: " << word_count << endl;

	for(int i = 0 ; i <= MAX_CHARACTERS-1; i++)
	{
		int count = characters_counts[i];
		if(count > 0)
		{
			cout << "'" << (char) i << "' --> ";
			cout << std::dec << count;
			cout << " " << endl;
		}
	}

}

Of course, there are limitations. For example, one is that this approach still bases a large portion of its logic on using spaces as word separators, so it doesn't consider punctuations as word separators. It only prints the ASCII characters that you use and probably countless other problems which isn't the point. 

On reflection, this is what practise is and why it's important. It's not about the triviality of the problem, it's the dynamic affordances that are provided by encountering problems.

Besides this, I set myself an arguably more interesting task recently, which was to figure out how to generate all the permutations of a series of digits. For example, if you had a combination lock with 6 digits, generate all the possible permutations.

To generate the Tth last digit for the nth permutation, this general formula can be used:

\[\lceil{\frac{n}{p^{t-1}}}\rceil - P\lceil{\frac{n}{p^t}}\rceil -P\]

where t is relative to the last digit to generate, i.e t=1 is the last digit, t=2 is the penultimate etc, and P is the number of possible values that this digit can have.

For example a 3-digit code where each digit can be A-D, ie 4 possibly sumbols per digit (p=4), would allow you to generate the 12th (P=12) permutation like this:

\[ \lceil{\frac{12}{4^{2}}}\rceil - 4\lceil{\frac{12}{4^3}}\rceil -4, \lceil{\frac{12}{4^{1}}}\rceil - 4\lceil{\frac{12}{4^2}}\rceil -4, \lceil{\frac{12}{4^{0}}}\rceil - P\lceil{\frac{12}{4^1}}\rceil -4 \]

This would result in the T=3, 2, 1 mapping to digit values: 1, 3, 4 which map to 1= A, 2=B, 3=C and 4=D. So the 12th permutation is ACD.

I wrote a program the verify this:

#include <cmath>
#include <iostream>
#include "permeate.h"

typedef unsigned long long int SIZE;

using std :: cout;
using std :: cin;
using std :: endl;

void print_n_permeation(SIZE n, int p, int num_terms, char* possibilities)
{
	cout << "n=" << n << ") " ;
		for(int t = num_terms; t >= 1; t--)
		{
			int a = (ceil(n/pow(p,t-1)));
			int b = p * ceil(n/pow(p,t));
			int c = p;
			int result = a - (b -c);
			cout << " " << possibilities[result-1] << " ";
		}
		cout << endl;
}

void print_all_permeations(SIZE total_permeations, int p, int num_terms, char* possibilities)
{
	for (int n = 1; n <= total_permeations; n++)
	{
		print_n_permeation(n, p, num_terms, possibilities);
	}
}

int main(int argc, char* argv[])
{
	int num_terms;
	int p;
	cout << "Enter the number of digits:";
	cin >> num_terms;
	cout << "Enter the number of possibilities per digit:";
	cin >> p;

	char* possibilities = new char[p](); // dynamic array in c++s

	for(int i = 0;i < p;i++)
	{
		cout << "Enter possibility #" << i << ": ";
		cin >> possibilities[i];
	}

	SIZE total_permeations = pow(p, num_terms); // num_terms aka num digits
	cout << "Total permeations:" << total_permeations << endl;

	
	SIZE n; // specific permeation
	cout << "Enter permeation to find (0=ALL, -1=None): ";

	while(cin >> n)
	{
		if(n == 0)
			print_all_permeations(total_permeations, p, num_terms, possibilities);	
		else if (n == -1)
			break;
		else
		{
			if( n <= total_permeations)
				print_n_permeation(n, p, num_terms, possibilities);
			else
				cout << "Invalid, must be between 1 and " << total_permeations << endl;
		}
	} 
}

Pretty neat I thought, but there are other ways to do this without calculation, such as Heap's algorithm that swaps characters iteratively. Interesting non the less!

Since, ISO27001, Machine-Learning and Game dev, I recently wanted to store some sensitive data (private key) in a string in C# and keep it in memory during the duration of an operation.

Due to the design of some 3rd party APIs that required a string representation of the private key, I decided on encrypting the string and only unencrypting it when I want to use it. In all other instances, the encrypted string would be copied or passed around, while the unencrypted string would not be.

I did some reading about System.Security.Cryptography's ProtectedMemory function, which allows you to encrypt a block of bytes of which needs to be a multiple of 16 bytes. The interesting thing about doing this is being able to encode the length of your sensitive string within the actual encrypted 16n byte block so that when you unencrypt that block, you can retrieve from it the length of the original string, and recover the original string. This is kind of what you do when you encode the length of a packet that you send down the network.

The implementation of an object that can encrypt a string, store it internally and decrypt upon request, is a ProtectedString:

using System;
using System.Security.Cryptography;
using System.Text;

namespace X.Y.Z
{
    /// <summary>
    /// Store string encrypted at rest.
    /// </summary>
    /// <remarks>You can copy this object freely</remarks>
    /// <remarks>Portable alternative to SecureString, using DPAPI</remarks>
    /// <remarks>Note SecureString is not recommended for new development</remarks>
    /// <remarks>https://docs.microsoft.com/en-us/dotnet/api/system.security.securestring</remarks>
    public class ProtectedString : IProtectedString
    {
       /// <summary>
       /// Secret area that is encrypted/decrypted
       /// </summary>
       private byte[] _secretData;

       private readonly object _lock = new();

       private bool IsProtected { get; set; }

       /// <summary>
       /// DPAPI access control for securing data
       /// </summary>
       private readonly MemoryProtectionScope _scope;

       /// <summary>
       /// Creates a ProtectedString
       /// </summary>
       /// <param name="sensitiveString">Sensitive string</param>
       /// <param name="scope">Scope of the protection</param>
       public ProtectedString(string sensitiveString = null,
                              MemoryProtectionScope scope = MemoryProtectionScope.SameProcess)
        {
            _scope = scope;

            // Store secret if provided and valid
            if(InputValid(sensitiveString))
                Set(sensitiveString);
        }

       private static bool InputValid(string sensitiveString)
       {
           return sensitiveString != null;
       }

       /// <inheritdoc />
        public void Set(string sensitiveString)
        {
            try
            {
                lock (_lock)
                {
                    if(!InputValid(sensitiveString))
                        throw new InvalidInputException();

                    // The secretData length should be a multiple of 16 bytes
                    
                    var secretDataLength = RoundUp(
                        sizeof(int) + // We will store the length of the
                                      // sensitiveString as the first sizeof(int) bytes in secretData
                        sensitiveString.Length, 16);

                    // Allocate array, all values set to \0 by .Net
                    _secretData = new byte[secretDataLength];

                    // Copy the length of the sensitiveString into the secretData
                    // first, starting at the first byte
                    BitConverter.GetBytes(sensitiveString.Length).CopyTo(_secretData, 0);

                    // Copy the sensitiveString itself after the bytes the above bytes
                    Encoding.ASCII.GetBytes(sensitiveString).CopyTo(_secretData, sizeof(int));

                    // Encrypt our encoded secretData using DPAPI
                    ProtectedMemory.Protect(_secretData, _scope);

                    IsProtected = true;
                }
            }
            catch (Exception e)
            {
                IsProtected = false;

                if (e is ProtectedStringException)
                    throw;

                throw new Exception("Unexpected error while storing data from protected memory");
            }
        }

        /// <inheritdoc />
        public string Get()
        {
            try
            {
                lock (_lock)
                {
                    if (!IsProtected)
                        throw new NotProtectedException();

                    // Decrypt secretData
                    ProtectedMemory.Unprotect(_secretData, _scope);

                    // Determine how long our sensitiveString was by reading the integer at byte 0
                    var secretLength = BitConverter.ToInt32(_secretData, 0);

                    // Read that many bytes to recover the original sensitiveString
                    var sensitiveString = Encoding.ASCII.GetString(_secretData, sizeof(int), secretLength);

                    // Re-protect secretData after retrieval
                    Set(sensitiveString);

                    // Return a reference to unprotected string.
                    return sensitiveString;
                }
            }
            catch (Exception e)
            {
                if (e is ProtectedStringException)
                    throw;

                throw new Exception("Unexpected error while retrieving data from protected memory");
            }
        }

        private static int RoundUp(int numToRound, int multiple)
        {
            if (multiple == 0)
                return numToRound;

            int remainder = numToRound % multiple;
            if (remainder == 0)
                return numToRound;

            return numToRound + multiple - remainder;
        }
    }
}

The question is if this is really useful at all from a security standpoint?

As soon as you unencrypt the contents, you get an unencrypted string back, and that string lives in memory and in theory, can be looked at by memory scanning. Also when that memory is freed (provided you don't have a reference to it), the garbage collector will free it but won't zero it out (securely clear it), so it'll be somewhere in memory, ...unencrypted.

Ultimately I never used this because of the reasons mentioned above, but it's still interesting...

Now, despite this, the above is still useful in some ways, provided you:

• a) only copy or store the protected string or pass it between functions.
• b) don't store the unencrypted string anywhere.

The other advantage is that the window of exposure of the unencrypted string is small (but it'll still get garbage collected), as you only unencrypt the ProtectedString when you want to use it, otherwise the secret is encrypted at rest.

Still, it doesn't help with the original problem of having unencrypted string copies lingering in system memory somewhere....