• Initializing a Ruby interpreter from libruby
• Calling Ruby methods and functions
• Registering methods or functions that will be called from Ruby code
• Converting data between the two Worlds (right now the most complex instance is the JSON one, which means that many complex Ruby types can’t be converted, but it is more than enough for passing data)
• Embedding native Haskell values that can be passed around in Ruby to the Haskell-provided external functions (I will use this for passing the Puppet catalog state around)

There are still a few things to do before releasing it :

• Making compilation a bit less dependant on the system. This will probably require quite a few flags in the cabal definition …
• Hunting for memory leaks. I am not sure how to do this with the GHC Runtime in the middle, and I do hope that ruby_finalize frees everything that is managed by the Ruby runtime. After all, restarting processes seems to be the only working garbage collection method for Ruby daemons …
• Writing stubs for the Puppet library methods that might be needed by templates. I would like to be able to support custom types and functions directly written in Ruby instead of Lua, but this will probably turn into a nightmare …
• Cleaning things up !

Here is a quick code preview :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

{-# LANGUAGE OverloadedStrings, OverloadedStrings #-}
module Main where

import Foreign.Ruby.Bindings
import Data.Aeson
import Data.Attoparsec.Number

-- this is an external function that will be executed from the Ruby interpreter
-- the first parameter to the function is probably some reference to some top object
-- my knowledge of ruby is close to nonexistent, so I can't say for sure ...
extfunc :: RValue -> RValue -> IO RValue
extfunc _ v = do
    -- deserialize the Ruby value into some JSON Value
    onv <- fromRuby v :: IO (Maybe Value)
    -- and display it
    print onv
    -- now let's create a JSON object containing all kind of data types
    let nv = object [ ("bigint" , Number (I 16518656116889898998656112323135664684684))
                    , ("int", Number (I 12))
                    , ("double", Number (D 0.123))
                    , ("null", "Null")
                    , ("string", String "string")
                    , ("true", Bool True)
                    , ("false", Bool False)
                    , ("array", toJSON ([1,2,3,4,5] :: [Int]))
                    , ("object", object [ ("k", String "v") ] )
                    ]
    -- turn it into Ruby values, and return this
    toRuby nv

-- this is the function that is called if everything was loaded properly
nextThings :: IO ()
nextThings = do
    -- turn the extfunc function into something that can be called by the Ruby interpreter
    myfunc <- mkRegistered2 extfunc
    -- and bind it to the global 'hsfunction' function
    rb_define_global_function "hsfunction" myfunc 1
    -- now call a method in the Ruby interpreter
    o <- safeMethodCall "MyClass" "testfunc" []
    case o of
        Right v -> (fromRuby v :: IO (Maybe Value)) >>= print
        Left r -> putStrLn r

main :: IO ()
main = do
    -- initialize stuff
    ruby_init
    ruby_init_loadpath
    -- and load "test.rb"
    s <- rb_load_protect "test.rb" 0
    if s == 0
        then nextThings
        else showError >>= putStrLn

And here is the ruby program, that calls our external function :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

class MyClass
    def self.testfunc
        hsfunction( [16588,
                    "qsqsd",
                    true,
                    { 'a' => 'b' },
                    :symbol,
                    0.432,
                    5611561561186918918918618789115616591891198189123165165889 ]
                ).each do |k,v|
            puts "#{k} => #{v} [#{v.class}]"
        end
        12
    end
end

And the output, showing that data is properly converted from either sides :

1
2
3
4
5
6
7
8
9
10
11

Just (Array (fromList [Number 16588,String "qsqsd",Bool True,Object fromList [("a",String "b")],String "symbol",Number 0.432,Number 5611561561186918918918618789115616591891198189123165165889]))
bigint => 16518656116889898998656112323135664684684 [Bignum]
int => 12 [Fixnum]
double => 0.123 [Float]
array => 12345 [Array]
true => true [TrueClass]
null => Null [String]
string => string [String]
object => kv [Hash]
false => false [FalseClass]
Just (Number 12)

EDIT: added link to the code.

PuppetDB code reworked

The PuppetDB code and API has been completely overhauled. It is now more generic : the resource collection and puppet query functions now work the same. Additionally, a PuppetDB stub has been created for testing use.

Better diagnostic facilities

As the main use of this library is to test stuff, the following features were added:

• Several error messages have been reworked so that they are more informative.
• A dumpvariables built-in function has been added. It just prints all known variables (and facts) to stdout, and can be quite handy.
• The “scope stack” description is stored with the resources. This turned out to be extremely useful when debugging resource names colisions or to find out where some resource is defined.

Here is an example, let’s say you do not remember which package installs the collectd package. Just run this :

1
2
3
4
5
6
7

» puppetresources . default.domain 'Package[collectd]'

package {
    "collectd": #"./modules/collectd/manifests/base.pp" (line 4, column 9) ["::","site::baseconfig","collectd","collectd::client","collectd::base"]
        ensure          => "installed",
        require         => [Class["collectd"], Class["collectd::base"], Class["collectd::client"], Class["site::baseconfig"]];
}

You now know exactly where the package resource is declared, and the list of “scopes” that have been traversed in order to do so. Note that this information is displayed when resources names collide.

Easier to setup

This library doesn’t depend from a newish bytestring anymore, and should build with the package provided with a GHC compiler of the 7.6.x serie.

This is not yet done, but I will certainly soon publish a debian-style repository of the compiled puppetresources binary. I am interested in suggestions for an automated building system.

Better testing

The testing API seems sufficient to write pretty strong tests, but would still benefit from a few more helper functions. The testing “daemon” has been reworked to use the new PuppetDB stub. It makes it possible to test complex interactions between hosts using the exported resource or PuppetDB query features.

Work in progress

I will probably lensify the code until I get a descent understanding of it.

I do not intend to work on Hiera emulation just yet, as I am probably the only user of this library for now and I do not use this feature.

One area of improvement would be to embed the ruby interpreter in the library. I am not sure how to do this, but as there are quite a few projects of lightweight interpreters sprouting from the earth, it might be possible in the near future. The only problem would be figuring out how to build a large C project with cabal.

Some other considerations

I recently ported the code from random.c to Haskell (here). This has been quite tedious, and is quite hard to read. This is an almost naive port of the code found in the Ruby interpreter, without the useless loop variables. For some reason, there are many loops like this :

1
2
3
4
5
6
7
8
9
10

i=1; j=0;
k = (N>key_length ? N : key_length);
for (; k; k--) {
    mt->state[i] = (mt->state[i] ^ ((mt->state[i-1] ^ (mt->state[i-1] >> 30)) * 1664525U))
        + init_key[j] + j; /* non linear */
    mt->state[i] &= 0xffffffffU; /* for WORDSIZE > 32 machines */
    i++; j++;
    if (i>=N) { mt->state[0] = mt->state[N-1]; i=1; }
    if (j>=key_length) j=0;
}

As you can see, the value of k is never used in the loop. I am not sure why the author didn’t go for something like :

1

for(i=1;i<k;i++) {

Anyway, the Haskell code is pretty bad, and will certainly only work for 64-bit builds. I am not sure how I should have written it. I suppose staying in the ST monad would have lead to nicer code, and I am open to suggestions.

I stuck with my grand mission of rewriting the handful of useful Ruby programs, and wrote a new package. I based almost everything around the excellent conduit abstraction. It has the following features:

• Haskell types for representing Logstash messages, along with the type-classes necessary for converting them from and to JSON
• An ElasticSearch conduit, using the bulk insert API
• A Redis source using the pipelining features of the hedis package, and a simple Redis sink
• A Logstash listener, based on the TCP listener from network-conduit, able to accept latin1 and UTF-8 messages at the same time
• A pair of “retrying” sinks, one using a Socket and the other establishing TCP connections. They are used for garanteed delivery of a whole ByteString segment, retrying to connect until it is sent (this is obviously useful for JSON messages)
• A few functions for handling bulk APIs in Conduits
• And finally, the coolest part, a few helper functions that will let you route between conduits !

The last part was made after a little discussion on the Haskell-Cafe mailing list. It is built with with stm-conduit, which already has a helper function for merging sources. This package introduces the other useful functionnality: the ability to “route” items coming from a source to several sinks. The main function, branchConduits, works by taking a Source, a routing function, and a Sink list. The routing function associates a (possibly empty) list of integers to every item coming from the Source. These integers directly map to the corresponding Sink, letting you define the routing policy.

The package includes a few examples of common tasks, all of them with acceptable runtime performance, such as :

• Moving messages from a TCP server to Redis
• Moving messages from Redis to Elasticsearch
• Routing messages between conduits

So if you need more control, or much better performance, than what you would get from Logstash, and you are not afraid to write (a lot of) code, please use this package and let me know what is missing and/or buggy!

Now, with PuppetDB, we have a simple and powerful way to create new effects, beyond what could be achieved with just exported resources (I believe it used to be possible before PuppetDB, but required black magic in the template files). I will demonstrate a typical use case, along with a sneak peak of the testing features that will appear in the next version of language-puppet.

Let’s say we have an HTTP proxy and several groups of servers acting as backends. You wish to be able to add servers to the pool just by running the agent on them. The site.pp should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

node 'proxy' {
    include haproxy
}

node 'back1' {
    haproxy::backend { $hostname:
        backend_type   => 'web',
        backend_server => $hostname,
        backend_port   => 80;
    }
}
node 'back2' {
    haproxy::backend { $hostname:
        backend_type   => 'web',
        backend_server => $hostname,
        backend_port   => 80;
    }
}

The haproxy::backend defines are empty, and the interesting part is in the haproxy class:

1
2
3
4
5
6
7
8
9
10
11

class haproxy {
    $backends = pdbresourcequery(
        ['and',
            ['=',['node','active'],true],
            ['=','type','Haproxy::Backend']
        ],'parameters')

    file { '/etc/haproxy/haproxy.cfg':
        content => template('haproxy/config.erb');
    }
}

The pdbresourcequery function comes from this excellent module, and has been included natively in language-puppet for a while. Its effect here is to fill the $backends variable with an array containing all resources that are of type Haproxy::Backend on any active node.

But now comes the complicated part: how are you supposed to write, and, more importantly, to test, the config.erb template ? As far as I know you can’t pull this off with puppet-rspec (and it is way too slow anyway). With the new testing API, you can write a simple program like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

module Main where

import qualified Data.Map as Map
import Control.Monad (void)
import Puppet.Testing
import Puppet.Interpreter.Types
import Facter

main :: IO ()
main = do
    qfunction <- testingDaemon Nothing "." allFacts
    void $ qfunction "back1"
    void $ qfunction "back2"
    (proxycatalog, _, _) <- qfunction "proxy"
    case Map.lookup ("file","/etc/haproxy/haproxy.cfg") proxycatalog of
        Nothing -> error "could not find config file"
        Just f  -> case Map.lookup "content" (rrparams f) of
                       Just (ResolvedString s) -> putStrLn s
                       _ -> error "could not find content"

Line by line, this program does:

• lines 1-10 : various headers
• line 11 : the catalog computing function is initialized, using the new testing system
• line 12 : the catalog for the node back1 is computed, and stored into the fake PuppetDB
• line 13 : same thing for back2
• line 14 : same thing for proxy, but we keep the final catalog this time
• lines 15-19 : the content of the /etc/haproxy/haproxy.cfg is displayed. This part is terrible and will be replaced by some helper soon.

The template groups the resources by their “backend_type” attribute, creates a backend block for each of them, and populates the blocks with the corresponding backends.

1
2
3
4
5
6
7

<%- backends = scope.lookupvar('haproxy::backends').group_by do |x| x["backend_type"] end -%>
<%- backends.each do |backendname, backends| -%>
backend <%= backendname %>
    <%- backends.each do |backend| -%>
        server <%=backend["backend_server"]%> <%=backend["backend_server"]%>:<%=backend["backend_port"]%>
    <%- end -%>
<%- end -%>

And the output is :

1
2
3

backend web
        server back2 back2:80
        server back1 back1:80

It works! With this feature, it will soon be possible to test and experiment with the most complex aspects of inter-node interactions.

• untar everything on your puppetmaster host

• run it with ./hspuppetmaster /etc/puppet +RTS -N (the first argument is the location of your puppet repository) as the puppet user

• modify your web server configuration to redirect requests for /production/catalog to 127.0.0.1:3000. This can be done in Apache by disabling the “High Performance mode” (wasted a few hours on this one), and adding something like that to your vhost configuration:

1

ProxyPass /production/catalog http://localhost:3000/production/catalog

There are quite a few things it will not do (such as updating the facts in PuppetDB), but you should experience much better catalog compilation times (I have right now catalogs that take more than the default two minutes timeout to compile with Puppet), sometimes much clearer error messages. As it is based on language-puppet, it is generally much more strict than Puppet. For example it will fail on any variable that cannot be resolved.

Please have a try and let me know how it worked for you (do use --noop!).

1
2
3

file :: [String] -> IO (Maybe String)
file [] = return Nothing
file (x:xs) = catch (fmap Just (withFile x ReadMode hGetContents)) (\_ -> file xs)

This should return Just the content of the first readable file in the parameter list, or Nothing if there are none, and should not leak any file descriptor. Now that I am finalizing the hspuppetmaster binary, I can use my library to (try to) compute catalogs on my production systems, using the standard puppet agent -t --noop. It turned out that the file function was misbehaving. Testing it in GHCi illustrates the problem:

1
2
3
4

> file ["/nothing"]
Nothing
> file ["/nothing", "/etc/hosts"]
Just ""

It seems to work fine, except all file contents are empty. This behavior seems to be common knowledge among Haskellers, and is due to the fact that the file descriptor is closed before the output is evaluated. This is pretty horrible (and surprising), and what is even worse is my solution:

1
2
3
4
5
6

file (x:xs) = catch
    (fmap Just (withFile x ReadMode (\fh -> do
                                        { y <- hGetContents fh
                                        ; evaluate (length y)
                                        ; return y })))
    ((\_ -> file xs) :: SomeException -> IO (Maybe String))

It is a bit longer because of the use of the non-deprecated version of catch, and because it explicitly forces evaluation of the output of hGetContents. This behavior was extremely surprising to me, and I would like to thank the people on #haskell for their help in devising a correct version (mine was along the lines of !y <- hGetContents, which worked for my simple examples, but was certain to fail at some point). This is the only IRC channel I know of where people are at the same time active, always helpful, and knowledgeable.

• Check a file syntax, and print what it thinks it is.
• Compute a whole catalog and display it in human readable format or JSON.
• Display details about a specific resource in a catalog, including special support for file contents (useful for debugging templates).
• And do the two previous items using facts and/or queried data from a real PuppetDB.

It is also fast enough to compute the catalogs of all your nodes in reasonable time, which opens possibilities you would not even dream of in the Ruby Puppet world. One of them is writing “integration tests” that let you check properties related to complex environmental interactions between hosts.

In order to facilitate this, I am in the process of writing a fully fledged testing API (it is still a bit lacking). It is strongly inspired by other testing APIs and should quickly evolve into something that is very easy to use. It is not the current focus (which is to replace an actual Puppet Master with my software), but I already implemented a test that is built in the puppetresources executable: it now checks that each source parameter in each file resources points to an actual file. This is a common error pattern to me (forgetting to create the file, mistyping its name, or placing it in the wrong directory) that has now disappeared.

Oh by the way, a new version is out ! Version 0.3.2 mainly changes the license, from GPL3 to BSD3. The choice was dictated by the sudden outburst of horribly uninteresting posts about licensing that has plagued Haskell-cafe during the last few hours. I hope this will end soon, or it will not be possible to differentiate this mailing list from that of Debian.

I will probably write a sample application on top of WARP and modify the configuration of my Puppetmaster to redirect catalog requests towards it. This means that there could be an efficient replacement to the Puppetmaster soon.

First of all, you will now get notifications when a resource is missing or when you have created cycles. There are still some bugs :

• The aliases are not taken into account.

• The relationship metaparameters on classes are ignored.

• With the released version, nothing is actually working. Sorry … I realized too late how broken it was. You might want to check github, or the updated binary packages.

This is the kind of error messages you will get when cycles are found :

1
2
3
4
5
6
7
8

puppetresources: The following cycles have been found:
  File[/a]
       -> File[/b] ["./manifests/site.pp" (line 557, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 556, column 9))
       -> File[/c] ["./manifests/site.pp" (line 558, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 557, column 9))
       -> File[/d] ["./manifests/site.pp" (line 559, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 558, column 9))
       -> File[/e] ["./manifests/site.pp" (line 560, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 559, column 9))
       -> File[/f] ["./manifests/site.pp" (line 561, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 560, column 9))
       -> File[/g] ["./manifests/site.pp" (line 562, column 9)] link is Just (RRequire,UNormal,"./manifests/site.pp" (line 561, column 9))

Please note how each resource and link position is displayed. This is a lot more verbose than what the vanilla Puppet spouts, but I believe it is also much more useful. Chasing after links defined as a resource chain gets old very fast.

This is the current error message when a relationship points to (or from) an unknown resource:

1

puppetresources: Unknown relation ("file","/b") -> ("file","/a") used at "./manifests/site.pp" (line 556, column 9) debug: (False,True,False,False)

This one is terrible, and will need to be reworked. There is still quite a bit of work, but I am fairly pleased at how everything seems to fold in place.

I started this project at the end of April, 7 months ago. The project is incredibly useful as it is right now, and I am confident I will be able to provide a robust and vastly more efficient puppetmaster, with a ton of helpful tools, before the first anniversary of this project.

v0.2.2

• New features
  • A few statistics are exported.

v0.2.1

• Bugs fixed
  • The defaults system was pretty much broken, it should be better now.
• New features
  • Basic testing framework started.
  • create_resources now supports the defaults system.
  • defined() function works for resource references.
  • in operator implemented for hashes.
  • Multithreading works.
  • The ruby <> daemon communication is now over ByteStrings.
  • The toRuby function has been optimized, doubling the overall speed for rendering complex catalogs.
  • Various internal changes.

A quick profiling revealed the following :

• The program couldn’t use more than a CPU.
• Almost all the time was spent computing templates.

I refactored the code a bit during lunch. The Daemon code was supposed to be multithreaded, and has been designed as such. The only problem was that I forgot to start several worker threads. This is now fixed.

The template computing time was reduced by spawning several threads for this task (see previous commit), and converting the String code to ByteString, using builders. The time is now mostly spent in the renderString function, defined as :

1
2

renderString :: String -> BB.Builder
renderString x = let !y = BB.stringUtf8 (show x) in y

I went with the definition in here, but it is much slower. If somebody has a better implementation, please let me know.

The ByteString move reduced the time it took to compute the catalogs to about 6 seconds, and parallelisation reduced it to about 2 seconds. It is a 5.5x speed upgrade for a few minutes of work, not too bad. The template generation still takes most of the running time (50% of the time is spent spawning and waiting for the Erb code, 20% preparing the inputs for the Ruby processes). Nice speedups could arise from parsing more complex Ruby expressions from the Haskell code, but it is not a priority now that the performance is acceptable.

You can grab the latest github repo to test it, but you will need a very recent bytestring, and you will need to fix the cabal file for luautils.

• language-puppet

• hsfacter

• puppetresources

A tentative test framework is included. It took a few minutes of hacking, and now the puppetresources application checks that file sources are valid. It doesn’t work for “private” stuff.

A huge bug was fixed: defaults finishing with a comma were interpreted as a function working on a hash! For example:

1

File { owner => 'root', }

Was seen as:

1

file({ 'owner' => 'root' })

Custom functions

Testing custom functions was almost impossible until now, as they required being compiled with the library to be used. It is now possible to include lua implementations of your functions alongside the ruby version.

It is a bit harder to write than the ruby versions as it can’t (for now) access anything from the “Puppet” side, such as facts, and you have to put up with the Lua syntax.

The other caveat is that it is stored right next to the ruby function right now, and Puppet tries to interpret it (and find syntax errors in it), so it isn’t very clean. I will move it for the next version, and will rewrite a few functions from Puppetlabs stdlib too.

From the implementation point of view, the difficult part was the fact that there were no instance for Data.Map. I wrote it and it is now part of the luautils package.

Custom types

As there was all the infrastructure to find files into subfolders, I also added a very weak custom type system. It will detect the file names in the usual places and will know these are valid Puppet types. It doesn’t perform any additional checks for now.

I might add a lua functionnality for fine grained verifications.

What’s next

The fabled dependency handling is not coming soon, but there is a few things that are already implemented and will be released next time:

• Support for the defined function. Just like the real thing it is parse order dependant.

• the create_resources function now accepts the third parameter (default (values).

• The in operator now works with hashes. I am not sure why the Puppet stdlib implements has_key …

Module structure

The whole interpretation process could be separated in the following parts (fitting the modules hierarchy):

• The parser. This is an ugly part, is it is the first part I wrote when I had no clue about monads, applicative functors, or that the do notation was just syntactic sugar.

• The interpreter, which will be discussed here. It is also an ugly part, but for other reasons.

• The “daemon”, which is a piece of code that glues everything together and provides a simple API for using them. It should scale on multiple cores too, but this hasn’t been tested yet.

• The PuppetDB communication facilities, that need to be reworked.

• The native types support, with checks about the consistency of their parameters.

• The plugin system, which currently handles user defined functions and types.

The catalog monad

Most functions in the catalog monad are of type CatalogMonad, which is just an error/state monad over IO :

1

type CatalogMonad = ErrorT String (StateT ScopeState IO)

The fact that IO is needed is unfortunate, but cannot really be avoided, as templates are computed and PuppetDB is queried during catalog evaluation. It would have been possible to return partially evaluated catalogs to an outer function that lives in IO and that would be responsible with handling this, but I thought it would just make things complicated.

Template computation is another matter. It is theorically possible to run it as a pure function (if you accept to only use a subset of the Ruby language), but it would require a full fledged Ruby interpreter. This is not an option here, and templates are computed by spawning a Ruby process every time. Now that a LUA interpreter is embarked, it would be possible to write lua versions of the templates and have them interpreted in the same process.

Achieving position independency with a single pass

The catalog computing function basically goes through the AST (following includes and defines) and tries to resolve everything it processes. The problem is that the Puppet language is supposed to be referentially transparent, especially according to the the language guide, which was the only documentation for quite a long time.

Puppet language is a declarative language, which means that its scoping and
assignment rules are somewhat different than a normal imperative language. The
primary difference is that you cannot change the value of a variable within a
single scope, because that would rely on order in the file to determine the
value of the variable. Order does not matter in a declarative language.

Reading this, I presumed that the position of a variable assignment, like everything else, did not matter. It turns out it does, but it is now a bit late to correct this.

In order to satisfy this false assumption, all data and resource types exist in two flavors. For example, with values we have Value and ResolvedValue. It would be a good idea to remove all this cruft right now, but I am not in a hurry to touch it, as it represents quite a bit of code and seems to work alright for now.

It presents several challenges, as values are supposed to be transformable into their resolved counterpart at any time, but many things are context dependant (such as variable scoping, local variable presence, variables in defines, etc.). It also doesn’t work at all (just like with Puppet) with control structures or templates that rely on not-yet-defined values.

If somebody knows what the “proper” way to do this is, I would be quite interested. I am almost certain this exists as it seems related to writing fast compilers, which is a subject that has certainly been explored by smart people.

Room for extension

Finally, the most important type, besides that of the state, is the type of the main function getCatalog. One can notice there are quite a few functions that must be passed along. The reason is that it should be possible to swap backends easily. An example that might be written shortly would be to give two implementations for the puppetDB function:

• A generalization of the current version, that queries a real puppetDB.

• An emulated puppetDB that is populated with other runs.

This would need passing around more functions (a function to query arbitrary values to start with, and a function to store the exported resources), but would be immensely useful. It would make it possible to write test suites that cover the whole node list, including exported resources between hosts.

Now that would be pretty cool, right ?

• New users will need to meddle with the source code to add their own functions.

• These functions will ne be easy to distribute.

• Everybody will have access to the custom functions used in the manifests I work with.

In order to make things a bit better, I used the Lua module in Haskell to provide a small scripting language for people to use. It might have been easier for them to just add a full ruby interpreter, but that would have been against my convictions and is probably pretty difficult. Note that this is not yet released.

Now that this is done, it might be possible to add custom types in the same way, as well as one of the lua templating systems. This will make it possible to get rid of the ruby dependency and largely increase the performances.

In the next few posts I will describe some of the inner workings of this module and implementation samples.

By the way, there are now links for the documentation.

It is not as cool as the compiled version because you lack the interactive features, but is still pretty powerfull and much easier to setup.

• Facts can be retrieved from PuppetDB.
• Exported resources can also be retrieved from PuppetDB.
• The resource query function from this module has been implemented.

This means you can now test all your higher order Puppet recipes. A typical use case is for the configuration of proxy servers. It gets really easy to declare something like this on a host:

1
2
3
4
5

@@backend { $::hostname:
    port => 1234,
    type => 'static_content',
    tag  => 'productionproxy';
}

Then you can get them in a hash with :

1
2
3
4
5

$h = pdbresourcequery(
        ['and',
            ['=', ['node','active'], true],
            ['=', 'type', 'Backend']]
        , 'parameters')

And use it in a template with :

1
2
3

<%
scope.lookupvar('myclass::h').sort.each ...
%>

This is the kind of things that drives people to Puppet. On the other hand, it quickly gets tricky to test. How are you supposed to check that your template will look the way you expect it to ?

Previously I used to just hardcode the result of the pdbresourcequery function during the testing phase, and use puppetresources to look at the rendered template. Now with PuppetDB integration you get the actual facts and exported resources that Puppet will use when computing a catalog.

So how do you use it ? It requires having access to a clear text HTTP port that answers PuppetDB queries. Typically, you will want to use a SSH tunnel if you test on your workstation, just like that:

1

$ ssh -L 8080:localhost:8080 your.puppet.master

Once this is done, you have the following options, depending on how you use the language-puppet library :

• With puppetresources, start with the new -r switch, like this :

1

$ puppetresources -r http://localhost:8080 ./puppet/ node.test

• With the puppetresources in interactive mode, initialize the Daemon with the initializedaemonWithPuppet function, like this :

1

> queryfunc <- initializedaemonWithPuppet (Just "http://localhost:8080") "./puppet/"

• With the Daemon API, modify the Prefs records so as to add the correct URL:

1
2

let prefs = genPrefs "./puppet/"
queryfunc <- initDaemon (prefs { puppetDBurl = Just "http://localhost:8080" })

If you wish to use the lower level APIs, you might want to check the new PuppetDB modules (unfortunately hackage doesn’t seem to generate documentation at the moment so I can’t link it). You might use the built-in PuppetDB query function, use your own or just mock their output.

There are probably bugs, and the URL to access to pdbresourcequery is currently hardcoded (won’t take long to makes it right), but this is a major milestone for this project.

First of all, quite a few bugs were fixed since 1.7.0, and a few nice features have been added. A lot of minor enhancements have been added now that the official language documentation is published. The behaviour of the library should now more closely match that of Puppet. The problem is that there is a huge difference in behaviour that might bite you.

When I started this project, all that was available was the language tutorial and the Puppet source code. As I explained already, I find it less troublesome to just rewrite it from scratch than to try to follow Ruby code. I recently wrote a pair of resource providers, so I can attest this still holds true. It seemed to me at that time that the various statements could be inserted at any place in a manifest and without changing its meaning.

Some effects would be undefined obviously (putting two conflicting defaults in the same class for example), but everything seemed doable with little effort. One of the early design decisions was to support this feature fully, except in conditionnals where I couldn’t see how to handle them efficiently.

The cost of this decision is that all data types have been doubled : one version for the “raw” or “unresolved” version, and one version for the “final” or “resolved” version. Everytime a statement is interpreted, it might be left in an unresolved state, which leads to all kind of performance and logic problems (for example, when you can’t resolve a variable you have to store some pointer to its scope to resolve it later).

It turns out that this was not needed to be Puppet-perfect, as Puppet doesn’t even attempt to do this for variable assignements. This means that every data types could be fully resolved when first found.

This means that the following code fragment will work as you might expect in language-puppet, but not in Puppet:

1
2

file { $filename: ensure => present; }
$filename = 'foo'

And while language-puppet is much faster than Puppet, despite spawning a ruby process for (almost) every template evaluation, it could have been even more. Also the internals would be much cleaner …

Anyway, here goes the changelog:

New features

• Amending attributes with a collector.
• Stdlib functions : chomp
• Resource pretty printer now aligns =>.
• Case statements with regexps.

Bugs fixed

• Various details have been modified since the official language documentation has been published.
• Better handling of collector conditions.
• Solves bug with interpolable strings that are not resolved when first found.

New features

• Fix bug with ‘<’ in the Erb parser !
• Assignments can now be any valid Puppet expression.
• Proper list of metaparameters.

Bugs fixed

• Quick resolution of boolean conditions.
• Start of the move to a real PCRE library.
• Function is_domain_name.
• New native types : file, zone_record, cron, exec, group, host, mount.

In order to facilitate this process, you can use the diffing feature of the puppetresources program. The following lines describe a typical session, where one would like to check that the set of resources described in the old catalogs for host oldhost somehow match that of newhost.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ ghci Main.hs

> queryold <- initializedaemon "/path/to/old/manifests"
> querynew <- initializedaemon "/path/to/new/manifests"
> oldcatalog <- queryold "oldhost.domain"
> querynew "newhost.domain" >>= diff oldcatalog
...
> querynew "newhost.domain" >>= diff oldcatalog
...
> querynew "newhost.domain" >>= diff oldcatalog
...
> querynew "newhost.domain" >>= diff oldcatalog
...

Note that there will almost always be a difference, as the list of classes is, for now, part of the catalog. On the other hand, the relationships between resources will not be checked as it is not (yet) handled by the tool.