Since Games, STL, Digital Forensics and a Fedora, I've been pushing forward, however, I'm pretty tired this weekend. I skipped my run into the office on Friday because I suspected that I was coming down with something and thought better of it.
I went to the gym yesterday and I felt a bit weak. The run in was a bit tough because the sun was blazing down, though I think that unlike most, I'm ok with running in the heat, still, it was hot under the sun.
I worked on my back and I had to pace myself because there were times that It was pretty tough going. I think because I'm not as regular as I used to be, I've not developed the resistance to hard training so when I do it, its a tough going. But I just need to adapt to my new routine.
I had to slow down during my bench rows, as I strained a little near the end of my sets and my lat pulldown wasn't as heavy as it normally is, though I kept my form which is important and this morning I can feel the areas of my back. The run back that day was also a little tough and I suffered somewhat but I'm used to that - so I continued through it. You got to be a little bit mental. maybe a lot.
Today I woke up and decided to do it all again, this time the run in wasn't that hot or tough but I can certainly feel my weight inhibit me. I'm trying to be in shape so that my running isn't so tough and I think its working, though perhaps just being tired made me feel sluggish.
I decided to do shoulders today, which is notable because I've strayed far from shoulders recently and I'm very happy with the results. I was able to articulate my shoulder through a full range of motion. It wasn't heavy but that's ok, I need to ensure my shoulder is strong and fully recovered. I guess I'll really determine the outcome of today from tomorrow. Again, much like yesterday I focused on full ranged motion and left my ego at the door. I ran back and felt pretty OK - a lot easier than yesterday.
After yesterday's session, I was a bit pooped and fell asleep after my shower - propped up on my bed headboard. That was a good little break because It allowed me to regain some energy and motivation.
I spend the rest of the day studying Law. Common and Civil Law and how the courts and the law are structured. It was pretty interesting. I also did some planning for my next assignment which is about Continuous Professional Development which I deferred until this semester, which gave me real problems last time so I'm coming into this with new eyes.
Had my weekly lecture on C programming on Tuesday which gets me home at around 22:00 and in bed around 23:00 which is a bit painful but necessary. I helped a colleague to understand the concepts inherent to C which are quite challenging at times. I didn't realise how loud we were talking and was asked by this one guy in the room, very loudly, to 'Stop that, its really annoying!". I was actually quite surprised, shocked really. The dude is also finding the work tough so I guess he was just really annoyed at not being able to get 'it'. We were going over functions and we've yet to reach arrays and pointers, which should blow his mind.
Anyway, made me realise how not used to being in a classroom environment I am.
Though I think I really helped my colleague. Boo-hoo to him. I said sorry and left it at that. Moving on.
Apart from this, I've been recently working with a colleague on a data science platform, well just the beginnings of it. It is pretty cool because its an application of my dataflowpipline project I recently started. The key requirement of the pipeline for a data transformation exercise are to put some data through the pipeline, run some anonymizers on it, and then run some feature extraction function on it.
I sat with him and we coded it up together based on his experience at Cern. I had to go away and re-code it because it only became clear to me that tone last aspect of the discussion (around building a delegate that builds a pipeline for just one item should be created). Sometimes my brain needs to sleep on it to work it out. Anyway, I worked it out(see below) and it was quite interesting and fun doing it(maybe I should get out more).
The key was to create a pipeline builder that would create a pipeline could take in any single item of data and anonymize it and then extract features from it, both these processes are all represented by custom Processes() that represent those stages of the pipeline. Each process is represented as a stage and each stage is represented as a function ie Func<T,T>. The goal, as mentioned previously, was ultimately to represent a delegate that would take a single item of data and construct a dedicated pipeline configured with the anonymisers stages and the feature extraction stages built in and then put it through the pipeline. Then the full transformation would use this generic typed delegate on all the items.
I came up with this prototype, which transforms a list of People objects in this case, but in theory, any objects can be used and the appropriate anonymizers and features can just be defined and passed in when creating the pipeline and it will bake those into the pipeline.
This is is the way you'd create a pipeline for a list of people. Just add Anonymizers and feature extractors:
// This is a generic builder where we can use any type of data
// and transform it into a array of, in this case, integers
// The features take a person and turn it into an inteter
// if there are 3 features, then 3 integers will result int[3]
// for one person passed into the pipeline
var builder = new Builder<Person, int>();
// This will take all the anonimizers, features and the data
// and transform all the items into items of feature results
var result2 = builder.Transform(items: listOfPersons2, anons: anons, features: features);
The Anonymizers are just a list of functions that transform the data somehow and the features extract features somehow and the result is a matrix of this transformation of all the items into this array of extracted features which then can be minded or run through an algorithm for analysis-a-la-data-science.
The anonymizers and features are defined here:
// Here are two anonymizers
// We'll represent them as stages in the pipeline
Func<Person, Person> Anon1 = (person) =>
{
person.Name = "Boop";
return person;
};
Func<Person, Person> Anon2= (person) =>
{
person.Age = person.Age + 5;
return person;
};
// These are the feature extractors
Func<Person, int> featureEx1= (person) => person.Age;
Func<Person, int> featureEx2 = (person) => person.Name.Length;
// turn the anonimizers and features into arrays of funcs
var features = new[] { featureEx1, featureEx2};
var anons = new[] { Anon1, Anon2 };
Actually providing multiple stages (anonymizers) in the dataflow pipeline library wasn't possible in v0.02 so I had to manually code a loop and this prompted me to release a new version of the library to do this.
It's 4 days old and has 22 unlucky souls using it.
Anyway, I'll put the full source code at the end of the post.
I popped into my little Italian dinner, located on the ground floor of the local shopping centre after the gym and had an Italian bean salad with Mediterranean veg, chicken, mushrooms, sun-dried tomatoes and avocado. Then I spent most of my day following that reading the Association Of Chief Police officer's Good practise guide to digital evidence, which can be found here: https://www.digital-detective.net/digital-forensics-documents/ACPO_Good_Practice_Guide_for_Digital_Evidence_v5.pdf which has been on my todo list, since last weekend. Quite a long read but I'm better informed now because of it.
So on the whole, another productive two days.
As is life.
My data aquisition and trigger POC source code as promised:
sing System;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Pipeline;
using static Pipeline.Pipeline;
namespace daq
{
// This is an example of the data we'll pass through the pipeline
// in order to transform it
class Person
{
public Person(string name, int age)
{
Name = name;
Age = age;
}
public string Name { get; set; }
public int Age { get; set; }
}
class Program
{
static void Main(string[] args)
{
// Some copies of the input data
Person input1 = new Person(name :"Stuart", age : 31);
Person input2 = new Person(name: "Stuart", age: 31);
// Here are two anonymizers
// We'll represent them as stages in the pipeline
Func<Person, Person> Anon1 = (person) =>
{
person.Name = "Boop";
return person;
};
Func<Person, Person> Anon2= (person) =>
{
person.Age = person.Age + 5;
return person;
};
// These are the feature extractors
Func<Person, int> featureEx1= (person) => person.Age;
Func<Person, int> featureEx2 = (person) => person.Name.Length;
// turn the anonimizers and features into arrays of funcs
var features = new[] { featureEx1, featureEx2};
var anons = new[] { Anon1, Anon2 };
// This will run all the features on a single Person
// This is specifically desinged to deal with Person objects
// So its not generic but we'll fix that later with a builder
Func<Person, Func<Person, int>[], int[]> featureVectorizationFunc = (person, featuresLst)
=> featuresLst.Select(fn=> fn(person)).ToArray();
// Prepare copies of the input as a series of Persons instead
// of just processing one person we want to process many
var listOfPersons = new Person[] {new Person("Jack", 29), new Person("stuart", 31), new Person("John", 45)};
var listOfPersons2 = new Person[] { new Person("Jack", 29), new Person("stuart", 31), new Person("John", 45) };
var listOfPersons3 = new Person[] { new Person("Jack", 29), new Person("stuart", 31), new Person("John", 45) };
// Examples of the data flow pipeline transformation process
// Pipe line of applying anonymiser in multi elements input before transforming result
// with custom function featureVectorization
// While this works, you need to provide each Anonimizer
var result1 = listOfPersons.Select(p => StartPipeline(() => p)
.Process(Anon1)
.Process(Anon2)
.ProcessAndTransform((person) => featureVectorizationFunc(person, features)).Result);
// This is a generic builder where we can use any type of data
// and transform it into a array of, in this case, integers
// The features take a person and turn it into an inteter
// if there are 3 features, then 3 integers will result int[3]
// for one person passed into the pipeline
var builder = new Builder<Person, int>();
// This will take all the anonimizers, features and the data
// and transform all the items into items of feature results
var result2 = builder.Transform(items: listOfPersons2, anons: anons, features: features);
// Example using the delegate which can process just one Item
var result21 = builder.SendThroughPipeline(input1, anons, features);
// Pipe line of applying anonymiser on one element of input before transforming result with custom function featureVectorization
int[] result3 = StartPipeline(() => input2)
.Process(Anon1)
.Process(Anon2)
.ProcessAndTransform((person) => featureVectorizationFunc(person, features)).Result;
}
}
/// <summary>
/// Turn input into a feature set, applying anonyizers on entry.
/// Fully Generic
/// </summary>
/// <typeparam name="T">Type of the items</typeparam>
/// <typeparam name="R">Type of the result matrix</typeparam>
public class Builder<T,R> where T : class
{
internal static Func<T, Func<T, R>[], R[]> featureVectorizationFunc = (person, featuresLst)
=> featuresLst.Select(fn => fn(person)).ToArray();
/// <summary>
/// function delegate that'd then be called with each object
/// </summary>
public Func<T, Func<T, T>[], Func<T, R>[], R[]> SendThroughPipeline = (T eachItem, Func<T, T>[] anons, Func<T, R>[] features)
=> StartPipeline(() => eachItem)
// Run the Anonimizers
.Process(i =>
{
// Really if Process could actually do this given an Func<IEnumerable<T>> of anonimizers that'd be much better than this
T prevResult = null;
for (int j = 0; j <anons.Length; j++)
{
prevResult = prevResult == null ? anons[j](i) : anons[j](prevResult);
}
return prevResult;
})
// Split item into feature results and put into matrix
.ProcessAndTransform((person) => featureVectorizationFunc(person, features)).Result;
public R[][] Transform(T[] items, Func<T, T>[] anons, Func<T, R>[] features)
=> items.Select(eachItem => SendThroughPipeline(eachItem, anons, features)).ToArray();
}
}