[Discuss] Interesting work in the industry?

[ccb at acm.org (Charlie Bennett)]

On Sun, 2012-07-22 at 08:42 -0400, Mark Woodward wrote:
> While I am currently employed at a pretty good company, I am constantly 
> getting recruitment emails. And they are all the same basic things, java 
> web sites or internet security. Isn't *anyone* doing anything 
> interesting anymore?

I'll bite.  Recruiter spam is uniformly useless.

It's not "the industry" its "the industries" and what you're going to do
there is a matter of personal taste and luck getting hooked up with the
right people.  I prefer to work for companies that use software to
leverage the sale of high-margin hardware or use software to provide a
service.  Being essentially a gear-head I've never really fit in with
"pure software" companies.

Maybe we can turn this into a discussion of 


   WHAT DOES THE INDUSTRY LOOK LIKE THROUGH YOUR EYES TODAY?


Me: 52 yo generalist UNIX programmer with system admin and release
engineering/SPLC skills.  I have a beard but it's not a "UNIX Beard".


After responding to an email on this list 7 years ago I've been
continuously employed by a company in Waltham that was a startup when I
joined but that got bought by Hitachi Data System in early 2007.  The
product is a "cloud-ready object store", meaning that it's a clustered
and somewhat highly available system.  It's built on Linux with a
boatload of Apache Java stuff in the front end.  The management
interface is Java JSP with some jQuery.  You can batch manage it over
HTTPS using XML.  Access to data is over HTTP/HTTPS/Webdav via Jetty or
via Linux Native NFSv3 or CIFS via Samba.  It can archive email sent to
it via SMTP.  It can be dumped to NDMP.  There can be as many as 80
nodes.  They can have local disk or they can pick up storage from a SAN.
We're talking any single cluster can have petabytes - billions of files
- under management.  It has all the bells and whistles needed for
Sarbanes-Oxley and all of your litigation process requirements.  It's
multi-tenant - you can carve it up and give private control to
subgroups.  It can inhale files or cough them back up at line rate over
1 GbE with 10 GbE on the horizon.

This is generally the kind of rig being built at the large storage
companies - Hitachi HCP, EMC/Isilon, HP.  To build something like this
you need...

Operating systems people: we run a Fedora derivative with special stuff
to sidestep the fact that nobody but big companies have access to SAN
gear.  We don't need Anaconda for install or dracut/plymouth/initramfs
for boot so we did our own.  We're multipathed with dm-crypt on ext4 so
a lot of the default disk stack behavior needs tweaked, etc....

C/C++ programmers: for close-to-the-iron backend stuff and high speed
networking.  Anything in our system that touches the OS or actual file
data is generally written in C and gets data from Java over sockets.
Our C programmers need sockets and pthreads and need to know pointers
and recursion - nothing all that fancy.

Java programmers: for the front end and most of the business logic.
This is the stuff Java is good for.  You can build it quickly but it's
not performance critical.  There is good work here, especially if your
application engine is massively multithreaded.  It's not all JSP.

Database guys: we've done a sharded PostgreSQL thing.  Putting a highly
available and redundant relational database in the middle of a large
architecture is both essential and daunting.  We're doing relational,
not mere SQL.  I'm sure goggling around can find a CJ Date diatribe that
can clarify.

Glue-gunners: Bash and Python.  We've sworn off Perl but use a fair
amount of awk.  You need these guys to make sure your application
starts, shuts down, rotates its logs, gets its ports open and closed,
etc.  Your OS guys will do some of this.

For every "developer" we have one half of a QA person and one half of a
QA automation developer.  We have more lines of Python test case code
than we do product code.  We have continuous integration builds running
over at least 4 branches at any given time with builds feeding a pool of
300-400 machines configured as more than 60 clustered systems running
test automation 24/7.

We have a tech writer.  And a book in our doc set that makes it very
clear that we use Busybox, etc. and how you can get a copy of the
source :-).

Each new project is done using what is effectively the Fred Brooks
"Surgical Team" model.  We work fast, we're nimble but most of us
consider "Agile" to be mostly bullshit.  Every release has had major new
functionality but we've not needed consultants to tell us how to make
our dates.  Shipping quality product that meets our commitments on time
is a very important feature of any release.  

We all have fully functional home lives.

Our product managers are not the licentious buffoons seen in a lot of
companies.



We've been hiring heavily and steadily since Day One, mostly because we
double the amount of stuff sold year over year, every year.

We hire for attitude and skill which is unfortunate since skills can be
taught.  Our management doesn't make offers to anyone that has even the
faintest whiff if edginess or high-maintenance, which is a bummer since
a lot of edgy and high-maintenance software developers are some of the
funniest people on earth.

We have something like 19 software development reqs open on all parts of
our stack, including QA, automation and lab system administration.





Outside my world, I know there are people working old-school iron,
mobile, industrial control, SaaS, IaaS, ISP, embedded, networking
equipment, educational, development tools and yes, Web Sites.

So what kind of mix of skills are needed to run your shop?  

How do you communicate that accurately to an outside recruiter???


ccb