I've been learning perl recently. Particualrly to re-write a program i wrote in both c# and java. The program is supposed to download stock prices form the internet in real time.
The biggest problem out there is that there aren't too many(actually I could only find one) free services that allow you to query stock information. Yahoo finance is the best one because its quite usable. There is a google service which they've discontinued. I guess, you'd get everything you wanted if you paid for it.
Anyway, the issue, once finding a service to query(yahoo) was figuring out which language to use to write it in. I naturally turned to C# because that's what i use day-to-day in my job. The issue with this was that this was actually written for my dad, who is quite forward thinking and doesn't use Windows. Sure, I guess one could use Mono on Linux but I'd not tried that recently. He also has a Mac(unix) so I figured Java would be a cross platform alternative. So I wrote it in Java. I also secretly wanted to write it in java.
The issue I had was that writing these solutions in a compiled language like c# or java means, well its a black box - you can't really extend it yourself without re-compiling it - something I'd do but perhaps no one else would like to do. So adding new features mean re-coding and re-compiling. Not the end of the world. I even thought about writing the software in C or C++ but though better of it(Although I love C). I thought better of it because of the amount of work which would be involved: you don't get advanced data structures in C by default, I guess you do with c++ but then you'd be compiling it and sealing up the solution again and it would be difficult to modify once its been deployed.
Actually, none of this really, hugely matters to me but because they are some reasons...and i only need one - i decided to do it in a scripting language. Perl is a scripting language with great support for lots of 'extras' that make it feel like a compiled language, which is usually has a host of libraries that make the work easier - things like parsing JSON, making HTTP requests etc.. Perl has this, and there was a big book on Perl in passing on a library book shelf, so that helped too.
I'll post the C# and Java versions just for fun later on but here is the Perl script (its highlighted in Ruby syntax - i dont have a perl highlighter at the moment)
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use Storable;
use JSON qw( decode_json );
use Scalar::Util qw(reftype);
use Parallel::Iterator qw( iterate_as_hash );
use Data::Dumper;
use URI::Encode qw(uri_encode uri_decode);
use Getopt::Std;
my %options=();
# -t 4, -d "delimiter" -o "outputfile.csv" -r "us" -l "en-gb" -v -x "exclude.csv" -l 5(co limit)
getopts("ht:d:o:r:l:vx:bl:", \%options);
my \(verbose = \)options{v};
my \(colimit = \)options{l};
if(\(verbose) {
foreach my \)opt(keys %options) {
print "\(opt = \)options{\(opt}\n";
}
}
if(\)options{h}){
print "./endofday.pl -t <numThreads> -d <delimiter> -o <ouputfile> -r <region> -l <language> -x <exclude file> -l <limit> -v\n";
exit(0);
}
sub ConvertCompanyToTicker {
my @args = @_;
my \(company = uri_encode(shift @args);
my \)region = shift @args || (\(options{r} || "us");
my \)lang = shift @args || (\(options{l} || "en-gb");
my \)json = getJson("http://d.yimg.com/aq/autoc?query=\(company®ion=\)region&lang=\(lang");
if (\)json) {
my \(jObj = decode_json(\)json);
my @queryResult = @{\(jObj->{'ResultSet'}{'Result'}};
for my \)var (@queryResult) {
return \(var->{symbol};
}
}
return undef;
}
sub ConvertTickerToStock {
my @args = @_;
my \)ticker = shift @args;
print "Live ConvertTickerToStockTicker: '\(ticker'\n";
my \)url = "https://query.yahooapis.com/v1/public/yql?q=".
"select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20(%22" .\(ticker.
"%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=";
my \)json = getJson(\(url);
if(\)json) {
my \(jObj = decode_json(\)json);
my \(queryResult = \)jObj->{'query'}{'results'}{'quote'};
return \(queryResult || undef;
}
return undef;
}
sub getJson {
my @args = @_;
my \)url = shift @args;
my \(req = HTTP::Request->new(GET => \)url);
\(req->header('content-type' => 'application/json');
my \)ua = LWP::UserAgent->new;
my \(resp = \)ua->request(\(req);
if (\)resp->is_success) {
return \(resp->decoded_content;
} else {
print "HTTP GET error code: ", \)resp->code, "\n";
print "HTTP GET error message: ", \(resp->message, "\n";
return undef;
}
}
#Global store of all tickers and their stock details
my %all;
#Read in Cache of ticker to company names to prevent relookup(expensive)
my \)cacheFileName = "co2tick.cache";
my \(progressCacheFileName = "progress.cache";
my \)haveTickerCache = -e \(cacheFileName;
my \)haveProgressCache = -e \(progressCacheFileName;
my %resolutionCache;
if(\)haveTickerCache) {
%resolutionCache = %{ retrieve(\(cacheFileName) };
}
if(\)haveProgressCache) {
%all = %{ retrieve(\(progressCacheFileName) };
}
\)SIG{'INT'} = sub {
store(\%all,\(progressCacheFileName);
exit 1;
};
# read in all the companies and exclude the ones in the exclude file
my @companies = <>;
my \)excludeFile = \(options{x};
if(\)excludeFile && -e \(excludeFile) {
print "using exclude file '\)excludeFile'\n" if(\(verbose);
open (EXCLUDE, "< \)excludeFile") or die "Can't open \(excludeFile for read: \)!";
my @lines = <EXCLUDE>;
my %exclude;
\(exclude{\)_} = undef foreach (@lines);
# exclude from companies those that are in exclude file
@companies = grep {not exists \(exclude{\)_}} @companies;
close EXCLUDE or die "Cannot close \(excludeFile: \)!";
}
# process companies
my \(lineCount = 0;
foreach my \)line(@companies) {
my \(company = \)line;
chomp \(company;
chop \)company;
next if !\(company;
my \)ticker;
# We're going to store company names with space to underscores so we can have one-worded companies in the cache
\(company =~ s/ /_/g;
\)ticker = \(resolutionCache{\)company};
# get a ticker
if(!\(ticker) {
\)company =~ s/_/ /g;
print "LIVE convertCompanyToTicker Company:'\(company': ";
\)ticker = ConvertCompanyToTicker(\(company);
print (\)ticker || "could not resolved to a ticker symbol.");
my \(addbadTickersToExcludeFile = \)options{b};
if(\(addbadTickersToExcludeFile && \)excludeFile && !\(ticker) {
#exclude bad tickers
open(my \)ex, '>>', \(excludeFile) or die "Could not open file '\)excludeFile' \(!";
print \)ex "\(company\n";
close \)ex;
}
print "\n";
next if(!\(ticker);
};
\)all{\(ticker} = undef if(!\)all{\(ticker});
# Put the multiword comany name in the hash that tracks all the results
\)company =~ s/ /_/g;
\(resolutionCache{\)company} = \(ticker;
last if (\)colimit && (\(lineCount++ == \)colimit));
}
#persist the cache of company to ticker hashes for future lookups...
store(\%resolutionCache,\(cacheFileName);
my \)numThreads = \(options{t} || 2;
my @columns;
my %output = iterate_as_hash({ workers => \)numThreads },\&ConvertTickerToStock, \%all);
%all = (%all, %output);
# Write all to CSV as output...
open(my \(csv, '>', \)options{o} || 'stocks.csv');
foreach my \(ticker (sort keys %all) {
my \)delim = \(options{d} || ";";
my \)stock = \(all{\)ticker};
#Get the first stocks values' order as default column order for all following stocks for csv format
if(!@columns) {
@columns = sort keys(%\(stock);
my @preferred = qw(Name Currency Ask Open PreviousClose PercentChange PriceBook Change DaysHigh DaysLow EarningsShare);
my @newColumnsWithout = grep {!/join("|",@preferred)/} @columns;
my @reOrder = (@preferred, @newColumnsWithout);
@columns = @reOrder;
print \)csv join(\(delim, @reOrder)."\n";
}
my @line;
foreach my \)key(@columns) {
my \(column = \)key;
my \(data = \)stock->{\(key} || "none";
push(@line, \)data);
}
print \(csv join(\)delim ,@line)."\n";
@line = undef;
}
close(\(csv);
unlink \)progressCacheFileName;
my \(end_run = time();
my \)start_run = \(^T;
my \)run_time = \(end_run - \)start_run;
print "Job took $run_time seconds\n";
This isn't as feature rich yet as the c#/java alternatives because its missing some features but the core is there and I'll add those features when I'm good and ready!
I've also added new features such as a progress cache and the ability to ctrl-c and abort and then continue later on. Also some other caching is in place to speed up the execution of the program. In fact, I'm dealing with an issue now where its TOO FAST(can you believe it!). I've querying the yahoo API using parallel threads*(http://search.cpan.org/~andya/Parallel-Iterator-1.00/lib/Parallel/Iterator.pm) and the yahoo service doesn't like that and starts issuing a HTTP 999 which probably means its thinks I'm doing a Denial of service on it! So I'll be doing some more work to make this better - this extensible and easily modifiable way to write code makes solutions progress very easily.
What I do find rather impressive about perl is that this is substantially shorter than the C# and java versions and a lot simpler.
I like Perl and I'd like to lean more. I have a university course on data visualisation which uses Python and even though i new this, Perl has an alluring thing about it - maybe its because its got such a long legacy in the Unix world and here is the real reason...I really like it's support for regular expressions and I've been yearning for a fast prototyping language that is quick, powerful and when it comes to text, is 2nd to none.
PS: I should probably add more comments but I love seeing the code so sparse! I'll probbaly comment more about this and perl syntax I've used in general moving forward.