Michael DeHaan created Ansible in 2012, but he was never very interested specifically into systems management technology and never really had an IT background of managing a server fleet himself. He was always more interested in software architecture, design, and programming — he tended to work on backend server applications, but not exclusively.

Why there was so much systems management in RTP (where Michael lived) was interesting. We can mostly credit IBM, which is amusing because this story starts with an entity that would later grow to own Ansible some 15 years later.

In the early 2000s, the Research Triangle Park (RTP) region of North Carolina was home to numerous systems management companies , likely due to IBM’s presence there since the 1960s. After the Dot Com crash cost Michael DeHaan a job offer at Nortel in 2001, he ended up at IBM working on RAID controller management software. IBM would later sell that division to Adaptec — following IBM’s pattern of spinning off parts of itself, such as it would later sell the laptop and server divisions to Lenovo.

While most of his time was spent with C, C++, and Java, to fill the time in long Adaptec status meetings, he learned Ruby and upgraded his Perl, using both while getting involved more and more with the company build system.

This period was also a time seeing how the management tools the company was writing were somewhat impractical at scale. There was no way to use them to manage a batch of servers. It was very much a "server as pets" kind of situation.

Had the Java work not been arduous, his Ruby interest would not have happened, then Python would not have happened (as quickly) and …​ so on. This would become key as these languages would grow in popularity to build IT automation tools and open source projects in the near future.

Right next door around the corner from Adaptec’s RTP office was a company called Integrian, which was working on early digital video solutions for recording what happens in front of police cars.

Integrian used Python extensively, including for GUI work. Michael joined Integrian working on utility GUI tools using wxWidgets in Python. It was a fantastic place to work for about six months before some VC actions decided to invest more in overseas locations.

Michael also met his friend Chris Church at Integrian — who would later help build the early Django backend for Ansible Tower and also (later) the first work on Ansible’s windows implementation.

Michael also learned a bit about how to manage engineering teams from how the engineering side of Integrian worked (and from counter-lessons from some other places later).

Michael ended up using his Python experience to join Red Hat.

Michael joined Red Hat around 2006, working on systems management tools in the "Emerging Technologies Group". "ET" was a large organization but the systems management team was about five people within it.

The ET team could mostly develop solutions to help customers without any product focus, however just a few cube rows over, Red Hat was also working on their commercial management stack, Satellite.

At Red Hat, after a request to make something like system-config-kickstart that could also handle virtualisation, Michael started Cobbler — most notably used as a PXE deployment framework. At the time, most PXE automation were handwritten by individual people at different companies, and not shared.

As Michael’s first open source project, Cobbler achieved significant adoption and was used in two top-500 supercomputers, chip design clusters, banks, render farms, and the infrastructure for Call of Duty’s server farms (a popular multiplayer game).

Michael’s manager left early on to join a startup called rPath, and he decided to more or less make the IRC channel for the Cobbler project (below) his boss, which would lay the foundations for how the community efforts at Ansible would later be run.

Uptake in Red Hat and CentOS shops was particularly large, and the CentOS community were particularly good contributors.

Cobbler was different than many other Red Hat "enterprise" open source projects in that there was a very strong community first focus including co-development, whereas in other projects it was more frequent that software was developed and the community was seen as a userbase. This ethos of not really revolving around the company was also common to the Fedora Project.

Cobbler started to indirectly compete with Red Hat’s closed-source Satellite, as people mostly needed an easy way to provision systems and get packages onto them. This underscored some early lessons about how GUI management software was often more tricky than managing things with scripts, version control, and automation. The ways people wanted to manage systems at scale was quickly changing.

Cobbler would also start to cross paths with a configuration management tool, Puppet, in very interesting ways, of which Ansible was an eventual alternative.

To avoid Satellite usage, Cobbler was often used in large installations with mrepo, created by Dag Wieërs or yum-utils’s reposync, where Seth Vidal was the lead developer. Both of these tools provided package mirroring solutions so users would not have to use GUI management tools like Red Hat’s own Satellite server. Both individuals would also later become very involved with Ansible (as detailed further below).

(It should be noted that RHEL customers using Cobbler vs Satellite still paid Red Hat and it helped them sell more operating system licenses, so free automation was still commercially useful to the company)

These tools (along with Puppet/CFEngine) would later sort of be rolled up into the idea of "DevOps" but none of these projects were really trying to align with that movement, culture, and term specifically.

Bare metal provisioning in Cobbler worked very well, but Cobbler would have probably benefited from using a database and having more of a "team" around it to help its development. Missing some potential to lead a team around it would later influence Michael towards leaving towards other places. Michael’s open source efforts sometimes felt a bit less robust to him than when he had time to work on something closed-source and commercial (often a lot of backends and API work) — all kind of a result of having to work in community contributions, interest, support, Q&A, and so on.

There was lots of multitasking.

Cobbler was named "Cobbler" because it makes machines boot, as a Cobbler is someone that makes shoes (and boots). Cobbler’s virtualisation helper piece was called "koan", for "kickstart over an network", which could also reprogram the boot loader of a system and reboot it so it would reinstall itself, either to specific configured profiles or to whatever was assigned on the cobbler server.

Cobbler assigned machines to particular roles usually, so this was an early example of trying to avoid configuration drift but applied to the bare metal OS install level. Common field practice was to bootstrap the machine just enough to run a configuration management tool, which was often Puppet.

To try to replicate Cobbler’s open-source success, Michael, Adrian Likins (of up2date fame), and Seth Vidal (author of yum) created Func — a project designed to fill the gaps between Puppet and Cobbler.

Puppet did configuration management well and was popular, but sometimes users needed to do something "now" and arbitrary, and it didn’t fit into config management ideas (which were more formal and declarative) at the time. This was not Puppet’s fault — it is just this other thing mostly didn’t exist. Uses of Func might include simple one-off tasks like installing software updates and rebooting servers, where puppet was about managing constant state. This could also include simply triggering puppet runs immediately rather than waiting for a periodic timer.

Greg DeKoenigsberg (a member of Red Hat’s community team and a former Fedora Project lead) suggested Michael, Adrian, and Seth get together to brainstorm new project ideas and this problem (above) is what they decided to work on. This the project "Func" was started, eventually backronym’d to "the Fedora Unified Network Controller". Func was a quasi-reference to Parliament Funkadelic, as chosen by Adrian.

Seth Vidal worked to port certmaster’s PKI from Puppet with Luke Kanies' approval. Among other parts, DeHaan wrote the parallelization feature "forkbomb" and the "hostglob" feature — both of which would become similar to features in Ansible. Adrian wrote other key important components and modules, and also maintained the project as Michael went back to work more on Cobbler and Seth went to work more on Fedora infrastructure and yum.

Ansible’s idea of modules being easy community development entry points also originated in Func. Compared with Ansible, Func ran a daemon on remote hosts that returned structured data vs using SSH. Some experience with users having PKI/SSL problems (NTP, DHCP, etc) would influence the need for configuration management systems to not rely on PKI systems, which was realized in Ansible.

Func was used in a few neat places including in some aspects of Fedora’s infrastructure systems, at Tumblr, and so on.

In many ways, Ansible’s "runner" (the piece underneath playbooks and the ad-hoc command line tool) was also an iteration on Func — but SSH instead of daemons.

At the same time, various Red Hat work was being applied around the AMQP message bus standard and an implementation called QPID, in partnership with various large customers. There was the idea of using a message bus instead for Func, but there were still complexities around usage and setup with messaging technology.

Such a message bus implementation was used in an attempt at a virtual appliance management and orchestration tool, Michael, Adrian, and Scott Seago were asked to write called "virt-factory". Red Hat would later go on to make other attempts at this with various projects and enterprise offerings, ultimately ending up with a version of OpenShift that was more or less a Kubernetes distribution.

In some ways, picking a specific message bus set up for Func would have created a bit of an adoption funnel - if somebody didn’t pick the message bus you picked, would they want to use your tool? This question would continue throughout Ansible’s development but it was fairly clear the idea of needing one was perhaps unimportant — Ansible would find out that people really didn’t want to reconfigure their entire server fleets all at once — that was often too dangerous! Thus it was seen that an over-emphasis on performance could be seductive, but it was a bit of an "engineer trap" because it’s such a fun problem to work on versus user surface areas that make more impacts.

If the message bus technology would have been a bit more complete at the time and easier to use with security features, Func would have probably used those, and Ansible may not have even used SSH. In the end, SSH ended up being the absolute right call and a key (along with language simplicity) to Ansible’s popularity and early adoption.

At Red Hat, harboring some feelings that Puppet content was a bit difficult to write, Michael wanted to build a config management layer on Func. He had named this idea Remote Rocket Surgery (there may have been a Wiki page once). But due to some internal politics in the group, building it was not an option. Basically there were people at Red Hat who did not want to compete with Puppet, for some reason.

This delay ended up being good as it provided a lot of idea and time (years) to think about what it could be like.

Around 2009, Red Hat underwent acquisitions and reorganization that resulted in the closure of the Emerging Technologies Group where everyone had to get "normal" jobs inside the company. DeHaan was asked to limit Cobbler work to 40% of his time (which he really liked working on for the community aspects), and there was a want to have him work on a merger between Red Hat Satellite Server and JBoss Operations Network that he was not interested in. It was clear that Cobbler could also not be maintained well with such a minimal time budget.

Finding a job that replicated that freedom from Red Hat’s "Emerging Technologies" group was difficult. Michael first chose an unnamed local place he did not like. Wanting to get out of that job, he briefly joined Puppet Labs (detailed below) for a short time.

As you can perhaps see, Puppet is starting to weave itself into this entire history and the need for Ansible is starting to emerge more strongly as time goes on.

Michael joined Puppet Labs as employee number 13 as a product manager who was also going to help out on the community (open source) side of things (based on the overlap with Puppet in perhaps 25-50% of Puppet install bases at the time), but this did not last long.

While the project was well established, the company was new. If Puppet had been ready for more structure at this stage and his travel not almost constant, he would have stayed — and Ansible would again not have happened. At this time there was still real no desire to resurrect the "Remote Rocket Surgery" idea.

This was however a good chance to observe some community structure to see how it was different from Cobbler and Func. Puppet was a bit more of a user community, than sharing development quite as much — there were some people very notably involved but it felt somewhat different. The company funded most of the core development and the community leaned more towards testing and advocacy and suggestions.

Ironically, the Puppet Labs experience really didn’t influence the course of Ansible too much but did confirm some prior feelings about the maintenance cost of a parser and PKI infrastructure. The team spent a lot of time on the "core" of the system, if anything, the idea that things needed to be internally simple was underscored.

Puppet was developing a GUI at the time, though there were also other community attempts to make alternatives.

(It is probably also worth pointing out that for a while many people would deploy and manage the operating system with Puppet, but would also use independent tools like capistrano to do deployments. Capistrano was SSH based, but better suited to "imperative" just once kind of tasks, and also did not need a daemon like Func. The need to learn two tools would become another driving force in shaping Ansible a bit later)

Michael needed a non-travelling job so he joined WebAssign, which was local to him and where knew a manager from the local photography group.

WebAssign was an powerful online homework solution that is infamously disliked by many thousands of college students. The backend was mostly all Perl, where Michael introduced some more recent "Modern" OO Perl modules and object oriented design concepts, working to increase code reuse.

While Michael was working on a feature that didn’t really see much adoption, he got to know the IT team pretty well and observed their deployment process.

The IT team would deploy software via scripts but could not adopt Puppet because it was very complicated. More interestingly, they had a rolling update topology using F5 load balancers, but did most of this process with a lot of manual interlocks, and the updates sometimes failed or human errors were inserted. Typically they would set up shop in a conference room to run the scripts through the chain of servers, and this would take possibly 30 minutes or several hours and failures did happen.

Observing this rolling update orchestration problem would be absolutely critical for Ansible’s success, inspiring the "serial" keyword and built-up rolling update support, which did not exist in any other configuration management tools at the time. Ansible was very much designed with this team always in mind, though not explicitly for them specifically. "Orchestration" kind of became a major keyword later in the company’s early history of explaining differences between more configuration-management focused tools.

WebAssign had about 100 servers in a pretty simple three tier architecture. Being able to work closer to IT also gave some better understanding of the potential of "DevOps" ideas to break down barriers. Developers not understanding metrics in production and the need to optimize SQL queries was a common pattern, but automation in production was also not made available to developers, so the stage and development environments were poor simulations of production.

Finally, a positive three-tier architecture experience there cemented a later desire to push back against very complex cloud systems, micro-services architectures, and so on — while Ansible also did really enable people to deploy those architectures as well.

Leaving WebAssign to escape "Scrum", Michael joined his old manager at rPath (where he had left from Red Hat), which was building virtual machine image/appliance creation solutions. Part of rPath’s product was really a better "enterprise" version of something like Packer (not created until 2013) but it was hard to use due to essentially having it’s own underlying operating system and packager. Due to marketing challenges and product complexity, the company did not survive.

Towards the end of that time, Michael was being asked to implement a Puppet integration to rPath that 'hid' Puppet language, which he absolutely knew how to do, but he was being told how to do it in a way that people would not like (using XML). Ultimately, he thought customers really wanted to just use Puppet if they were going to use Puppet.

While this solution was wrong, it became somewhat apparent that if he did not "solve" Puppet, Puppet was going to keep following him around at various systems management jobs which were most of what existed in the RTP area.

Amidst this frustration (XML!), Michael also missed the fun community aspects of running open-source projects and designing more software on his own.

Looking to more or less recreate the positive parts of the Cobbler experience, Michael started Ansible on nights and weekends.

(This is sort of a prologue to how things got started, and is when the decision that Ansible could be a company happened — the early history of the project occurred in parallel and is detailed below)

When rPath closed, Michael joined Cisco to write Python code for OpenStack, though this really didn’t happen. He got caught up in an endless Puppet effort to write content to automate OpenStack, and the whole team would restart working on the next version of OpenStack every time a version came out. This effort involved almost a dozen people! It was very complicated and ironic. (It also used Cobbler for PXE, which was there before Michael joined).

Because everyone on the team was scattered between different offices, it wasn’t a particularly great place to come into work.

Since he wanted to write code and not IT automation content, Ansible seemed pretty appealing now — could that be a job? While Ansible was growing in popularity quickly at the time, Cisco was not interested in it, since they had a strategic venture investment in Puppet.

The industry needed an escape not from Puppet itself, but from what consultants were doing with Puppet — or the fact that people needed consultants, or teams of 12 people to write Puppet.

This is an important understanding — the Puppet tool wasn’t the problem but what the tool got people to do tend to do with it. Tools however influence what kinds of tools people create with them. This experience would guide early Ansible development very much — the desire for automation content to be easy to read, for the most obvious paths to be the simple ones.

Aspects of "the Zen of Python" are influences in this.