You are here
Building a better, bigger TurnKey library
This blog post started out as a response to a private email discussion between me and contributing developer Eric (tssgery) whom recently developed TurnKey Observium, one of TurnKey's top ten downloads. Observium is very popular for a newly hatched appliance. It has the same number of downloads as Redmine, which we've supported for a few years now!
Eric wrote:
TKL has two big issues as I see it:
- Marketing. You're the best kept secret of the internet.
- Testing. Too many appliance/products to be supported.
Eric is right on the both counts.
-
TurnKey is severely undermarketed. We've been pouring all of our time and energy into development. I'm pretty sure there's a huge community of potential users that would find TurnKey very useful - except they don't know about it, or their cloud provider doesn't offer it as a deployment option, etc.
-
There's definitely room for improvement on the QA / testing front, as evidenced by the recent issue with TurnKey ClipBucket.
Testing is not only the limiting factor in the quality of the appliance library, but also its size. If it comes to that, and often it does, then like Eric I'd rather see TurnKey do a better job on fewer appliance solutions than an increasingly worse job on an ever expanding roster of solutions.
Luckily, I believe TurnKey has more levers to fiddle with than just the number of solutions in the library. Here are our main ideas on how it might be possible to improve the quality of the library while growing, rather than shrinking the number of supported solutions:
-
More community involvement: development labour is the key bottleneck, so it's very important we get more developers from the community on board. For this to happen we not only have to do a better job of encouraging community development but also make it as easy as possible for developers to get involved. Which brings me to the next points...
-
TKLDev as a cloud service: improve the build infrastructure further so that it is even easier for community developers to build and test up-to-date versions of new appliances. TKLDev is great, but I think we can do better. For example, one idea we've been toying with is turning TKLDev into a cloud service and tying it together with community repos on GitHub.
-
Built-in regression testing: build automated regression testing into the TurnKey build infrastructure. One of the "secret" reasons we've been working on deploying TurnKey on top of LXC is that it will make it easier for us to implement this.
Imagine if instead of manually testing appliances you just run:
make test
TKLDev could leverage LXC to run an instance of the built appliance, check that all the ports are open, access the web applications and make sure they are working (e.g., return 200, non-zero pages, no "error" or "exception" in the HTML, etc.) and use an automatic tool to screenshots a human could scan for errors in quick batches.
The design is pretty straightforward. Just extend how fab works with the other elements in the product design (e.g., conf, overlay, removelists).
In each product tests.d would contain a bunch of test scripts that would be run against appliance inside a container. For code re-use purposes these scripts would get executed with common/tests/bin in their PATH.
Similar to other elements instead of implementing a test in tests.d we could just point to a test via COMMON_TESTS variable, or to a group of common tests (e.g., COMMON_TESTS := turnkey.d in turnkey.mk) such as checking that all the ports in WEBMIN_FW_TCP_INCOMING are open.
What do you think?
Comments
I'll respond :)
I think there are more levers and I agree with the ones you have identified but don't think we'll be able to get there.
I am all for built in regression testing. My day job is running a couple of development efforts and the way we have achieved agility, quality, and predictability is to make sure we have a great suite of regression tests. We run ours nightly (or on demand). The major issue I see here is that for the tests to be effective, they have to excercise the functionality of the software. In the case of clipbucket, they'd need to upload a video, ensure it got converted successfully, and that it was available for viewing. This pre-supposes that the person writing the tests has a good working knowledge of the software and that it's automatable. I am not convinced we have that expertise on most applainces today.
TKLDev as a cloud service might be an interesting approach if it makes it easier for people to test their contributions. As an example, if I could fork a TKL appliance on github and submit my repo for building and testing... I'd be all over it. This is the mode that SuSE Studio Online uses and it's fantastic.
More community involvement is where we have the biggest issue. In my mind, we're in a catch-22. If we start marketing well and get a lot of new users, I am afraid they'll hit quality issues and leave which means no increased community. In my view, the communicty involvement today is mediocre. I'd estimate that there are less than 10 people who frequent the support community and lend assistance. What is going to change such that our current situation changes? Will people suddenly start writing tests for appliances? Will more people suddenly start helping Jeremy out on the boards? Will people start volunteering their time to be a community maintainer for one or more appliances? I have my doubts based upon the past few years of using TKL.
If it were my baby, I'd probably seperate the appliances into the following categories:
At the start, most applainces should be classified as "Best Effort" and we (the TKL developers and community) should look to move as many of them as viable to "Community Supported". This can be accomplished by actively seeking community members and marketing. As an example, maybe someone from the Redmine community would be willing to pick up maintenance of that appliance as it eases their deployment/configuration support. The same for clipbucket or other complex software.
We cannot, though, just assume that we'll get more community involvement to write tests, run tests, develop fixes, and support appliances without changing something fundamental. I see no evidence that will happen.
Making gradual, incremental progress
We'll never get "there" in terms of quality if the aim is perfection because perfection is an unattainable, moving target.
A more realistic aim would be to make gradual progress towards this goal and we're already thinking on parallel lines to the ideas you're suggesting (e.g., community developed apps), though some important pieces of the puzzle are still missing for this to happen.
Regression testing: anything is better than nothing. The ideal "regression test" would be a team of human experts testing the ins and outs of the software at every revision and so such a good job that nobody ever has to report anything to the issue tracker. That probably won't ever happen BUT identifying and quickly fixing the issues identified by the user community would be a good start. "With enough eyes, all bugs are shallow".
Automated regression tests can start out simple and gradually expand as we learn more about what tends to fail and how to catch it.
Lowering the bar on creating and testing fixes: Improving how easy it is to push out new appliances, both for us and the community will be a big help in accelerating the pace and frequency of development. Also, quick fixes encourage contributors to report bugs. It's empowering. Slow fixes discourage engagement.
More community involvement: I agree this isn't going to be easy and that significant progress is unlikely to happen overnight. On the flip side, with the right tools it doesn't take very many people to make a big difference (I.e., "Give me a big enough lever and I'll move the world").
I think it would be a good idea to attack the problem on multiple levels. Make the tools more powerful so we can squeeze out more productivity out of each hour of development, lower the bar on using those tools (e.g., TKLDev as a cloud service, training sessions), promote and celebrate the contributions of community developers such as yourself, run development contests, etc.
Also, it's important to remember we're not developing a new kernel here.
With the right tools the bar to contributing is lower than with most other open source projects. For most things you don't need programming skills, just integration level expertise. You can get a very satisfying amount of "bang" for your "buck". It's somewhere between programming and maintaining a wiki documentation page.
I think most people who have the skills and inclination to contribute aren't doing so because TurnKey isn't even on their radar. Promoting TurnKey doesn't necessarily mean promoting only to potential users. Perhaps just as important would be to promote TurnKey to potential developers by engaging and building relationships with more of the open source community.
I think most people who have
This, to me, is the key to greater community involvement and I agree 100%.
Great discussion guys!
I love this discussion and am pleased to see it going on. I agree with most of what has been said so I guess this is a bit of a 'me too' type post, but one thing I really wanted to support was your suggestion Eric for the appliance 'support levels' (I guess for want of a better phrase). I think that is a great idea and is something of a middle ground between limiting the number of appliances available through TKL and "having as many appliances as we can"...
Also I think that promoting TKL as something of a 'development partner' with other software developers holds huge potential. Seriously I don't think that there is anything not to like about that approach... As a software developer, having the facility to 'farm out' the work of maintaining (at an OS level), hosting and distributing of your software (in the form of a 'ready to run' TKL appliance) would surely be an apealing option!? And from a TKL perspective great to have a software developer 'onside' to support their software would be fantastic... I know that in the past this has almost happened a couple of times (ok at least once anyway...), but because all the pieces of the TKL infrastructure have not been in place and TKL wasn't responsive enough, it didn't happen and as such it died a natural death...
Reaching out to developers
Yes! Reaching out to application developers and getting them on board is something we are only now starting to think about in earnest. Up until last year we had some major gaps (e.g., lack of amd64 support and a release based on an old version of Debian) that we needed to close urgently before we could make a compelling case for partnering up with TurnKey. I think we can make a much better case for this now.
TKLX Mark II - The Community
TKLX appears to be at a juncture as the Cloud moves from bleeding-edge to leading-edge.
There is some tidying up of the TKLX Mark I version that makes sense and I agree with Eric and Liraz that categorisation, quality and LXC benefits exploitation is a reasonable target, but...what next?
I also agree with Eric's observation that if an active community has not formed already what is going to trigger a change in that situation? Me...I'm an ex-developer / R&D manager re-skilling. I'm still behind you guys in tech contribution skills so my lack of contribution at present is due to lack of time and skill level (I'm working on it). I can provide marketing / strategy suggestions though.
Is TKLX low adoption related to insufficient marketing or competitive alternatives?
Bitnami forums are segmented by appliance while TKLX is not. Changing the format might help a bit but the appliance community would still be limited to, well, the size of the appliance community.
Rasmus's Redmine puzzle as to why open source communities would not welcome the marketing opportunity associated with their application in an appliance has a number of possible explanations.
1) TKLX Appliance Competition. In the case of Redmine they have a Bitnami link next to the TKLX link. Bitnami monopolises the AWS marketplace first page for infrastructure solutions and shows up most places you look. Even though it is NOT open-source, Bitnami is claiming 1m downloads / month up from 100K downloads / month last year (http://techcrunch.com/2013/03/25/y-combinator-company-bitnami-makes-deep... ). Free commercially marketed appliances with proprietary packaging environment seems to be beating out free open-source word-of-mouth marketed appliances with open source packaging environment...go figure.
Type in the following three words in Google search: debian appliance aws
You get a Google adword directing people to the Debian supplied appliance on the AWS marketplace. Same price as TKLX core based on Debian. Is TKLX Debian appliance a better appliance than that provided by Debian! Tough to market that angle. TKLX is smaller but is that a real selling feature in a world of cheap storage! TKLX has bundled toolset and auto-config. Still a tough marketing pitch.
2) TKLX SAS Competition. In the case of open source communities such as SugarCRM...they make their money the same way TKLX does so they are competitors at the marketing level. No reason to promote TKLX. Interestingly if you search for turnkey linux on SugarCRM site you find a company named rpath offering a "turnkey linux" in the Sugar CRM marketplace. It would be interesting to have access to the TKLX download numbers and cross-match with the applications vendors who compete with TKLX. I would bet all the big names are TKLX hub competitors (Wordpress, Drupal, etc).
3) TKLX Hub Competition. AWS, Bitnami (possibly ylastic, scalr, rightscale, enstratus).
Bitnami with Bitrock as a steady investor and a recent $100K angel investment to expand IaaS base is switching from being an appliance supplier to an appstore supplier. Is the writing on the wall for the independent "appliance hub" vendor or is this going to be a boon for TKLX if the Bitnami appliance user-base becomes disenfranchised!
3) TKLX Technical Competition. People become involved in open source communties for social, commercial, technical interest and other reasons. Technical interest is a large driver though and there are a number of projects that I believe compete with TKLX for contrbutors.
Hub - Rehat CloudForms (some apache cloud contributors moved to this project -e.g. deltacloud) will opensource mid 2014 offering a cross-cloud management console based on ManageIQ technology. Redhat has also
TKLDEV - Puppet / Chef / Saltstack / Ansible offer a DevOps dream toy.
TKLBAM - Duplicati, Owncloud, DuraCloud offer a more generalised model for data backup & synch that has direct consumer appeal.
LXC / Docker - TKLX is not alone in recognising the potential. Redhat sees 2014 as the year of the Linux Container and has also partnered with Docker. http://www.redhat.com/about/news/archive/2013/12/15-infrastructure-predi...
Competition Summary:
Lots of competition for TKLX. So...
TKLX Mark II?
I think that Alon and Liraz need to review the hub-centric revenue model for TKLX. The cloud market has shifted and is shifting further and if TKLX usage drops below critical mass they may not be able to continue to contribute as much as they have in the past. Open-source communities live and die on the long term commitment of the founders. Alon & Liraz are the core of TKLX.
With a Cloudforms / Docker / Saltstack / Owncloud combination available by end of 2014 and Bitnami moving to be an appstore provider in recognition of the increased demand for mobile App back-ends, I think that TKLX needs to revisit its vision and objective to remain relevant through into 2015 and beyond and review it's hub-centric revenue model. No commercial model is impervious to competition...ask Blackberry designer RIM.
TKLX Mark II Options:
Option 1) Do nothing.
Keep tweaking the Mark I model. Some regression testing, some LXC improvements, some additional marketing
Community Contributions: Same as today.
Risks: Community remains the same as today.
Future Potential: Low. No real change.
Option 2) Follow Bitnami lead
Switch from a hub / appliance revenue stream to appstore revenue stream and potentially opensource your hub as a differentiator (proprietary hubs could take a serious hit if Cloudforms hits the mark).
Community Contributions: Same as today.
Risks: Community remains the same as today.
Future Potential: Low. Same model as before just different flavour.
Option 3) Go Mobile (my vote)
Partner with Phonegap and extend the TKLX philosophy to mobile app build & deployments. It leverages the TKLX Mark I investment.
Community Contributions: Same as today for the server but the size of the community would broaden significantly. I would expect the active nature of the app community to drive increased contributions to the server appliance side.
Risks: Lower than today. TKLX would move into a league of it's own. No Bitnami to market against. Some commercial variation to the current AWS-centric hub model required but incremental.
Just one man's opinion :-)
Cheers,
Tim (Managing Director - OnePressTech)
Hi Tim, I responded in more
Hi Tim,
I responded in more detail in our email discussion but I figured it would be good to put some of that back on the blog.
Market perspective: It's clear from your post you seem to know TurnKey's market better than we know TurnKey's market. We actually haven't been paying that much attention to our competition lately. That might sound crazy and it is! Which is another reason your outsider's perspective is so valuable.
For most of TurnKey's existence we didn't really think about it as a business. More of an open source project with a little business on the side to keep it financially sustainable. We didn't start TurnKey to get rich.
If we could have just ignored money altogether and had complete freedom to work on challenging and interesting open source orojects that help as many people as possible that would have been our ideal.
But reality intervened and we've been pay reluctant attention to the business side because there are bills to pay and because we'd like to be able to afford to fund as many talented people as possible to work on open source full time like us.
Lack of marketing: Unfortunately at the moment we're spread way too thin and up until recently have harbored some very naive notions (delusions?) about how users will discover the good work we're doing without us shouting too loud. Ha!
Business model: You raise some good points. We're exploring a few ideas for extending TurnKey business model well beyond cloud deployment. We're not sleepwalking through the changes in the industry but we're also not running amock in panic.
The important thing is that TurnKey's userbase is growing at a healthy rate. People are finding value if the app library. There are many untapped opportunities to add value to TurnKey users. We currently only provide a rough starting point for most users's need. There's an opportunity for more add-on services (e.g., like TKLBAM) that will bridge the gap between what you have and what you want. Either by developing solutions to those problems ourselves or partnering with third parties.
In other words, if someone is already using TurnKey and they are in the market for various commercial services native TurnKey services get a fair shot, especially high-quality well tested well-integrative native add-on services.
Some add-on services only require more engineering time to get off the ground. Others (e.g., support) only become viable with economies of scale and don't really make sense until you reach critical mass. That's another reason why we will be making an effort in the next couple of years to distribute TurnKey as far and wide as possible.
Rebranding: that's an interesting thing to consider. We're not really marketing people so that's not something we've given much thought to.
I guess we probably wouldn't want to make a move like without somehow testing first that it would make a meaningful difference. The TurnKey Linux brand may or may not be much, but it's all we have. I think to seriously consider doing something like that we would need better proof that throwing away the brand we've created and starting from scratch will do more good than harm.
We do have a marketing problem, but I think it's mostly the total absence of marketing rather than a specific branding problem.
Premature optimization is the root of all evil. There's definitely room for improvement in many areas but I tend to think of this sort of thing the same way I think about software optimization. I want to know where the performance bottlenecks are first before I spend too much energy into optimizing.
Mobile drive for TKLX Mark II: I'm not really familiar with this market. I need to research it more deeply.
Cool. Just thought I would add some options to the soup:-)
I look forward to your conclusion :-)
Cheers,
Tim (Managing Director - OnePressTech)
I'm actually writing you
I'm actually writing you another in-depth email reply. It's just taking forever.
Sounds like a good idea
This answers the question I
This answers the question I posted over on Tracker about the long term plans for TKLX testing. For most of last year I was working on plans to launch a small business and deciding what kind of infrastructure was needed to support the DevOps style organization I had in mind. I've pretty much abandoned those plans as it was hard to figure out how to make a profit serving the intended market, churches and non-profits. Still I feel there is a need for a suite of apps configured to work together implementing, DevOps for a small software shop. My dream team was GitLab/LXC/Ansible/Jenkins/Selenium+PHPunit.
Information is free, knowledge is acquired, but wisdom is earned.
Pages
Add new comment