After adding the cluster to ArgoCD as normal, take note of the user ARN used in the cluster’s configuration. Found in the users section of the kubectl configuration for the cluster.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <ca-cert>
    server: https://....eks.amazonaws.com
  name: https://....eks.amazonaws.com
contexts:
- context
    cluster: https://....eks.amazonaws.com
    namespace: argocd
    user: https://....eks.amazonaws.com
  name: https://....eks.amazonaws.com
current-context: https://....eks.amazonaws.com
kind: Config
preferences: {}
users:
- name: https://....eks.amazonaws.com
  user:
   exec:
     apiVersion: client.authentication.k8s.io/v1beta1
     args:
     - aws
     - --cluster-name
     - <cluster-name>
     - --role-arn
     - arn:aws:eks:<aws-region>:<account-id>:cluster/<cluster-name> # this is the user ARN to take note of
     command:i argocd-k8s-auth
     env: null
     interactiveMode: Never
     provideClusterInfo: false

Then add the aws-access-key-id and aws-secret-access-key for that user to both the argocd server and application contoller services. For example, in my case I’m making use of helm to install/setup ArgoCD, so the values.yaml includes the following configuration changes;

...
controler:
   ...
  env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-cluster-user
        key: aws-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-cluster-user
        key: aws-secret-access-key
  ...
server:
  ...
  env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-cluster-user
        key: aws-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-cluster-user
        key: aws-secret-access-key
  ...

See ArgoCD FAQs for some troubleshooting help.

class A():
  async def async_init(self):
    await self.calc()

    return self

  def __await__(self):
    return self.async_init().__await__()

  async def calc(self):
    return 1 + 1

A full example:

#!/usr/bin/python
import asyncio

class A():
    def __init__(self):
        self.sum = await self.calc()
    
    async def async_init(self):
        self.sum = await self.calc()

        return self

    def __await__(self):
        return self.async_init().__await__()

    async def calc(self):
        return 1 + 1

async def main():
    a = await A().calc()

    print(a)


if __name__ == "__main__":
    asyncio.run(main())

Step 1: Add [[source]] to Pipfile

As usual you’ll want to add the relevant [[source]] information to your Pipfile

[[source]]
name = "gcp"
url = "https://oauth2access@<location>-python.pkg.dev/<project>/<repo-name>/simple"
verify_ssl = true

The oauth2access part can actually be any user name, GCP docs do refer to _json_key_base64 at times.

Step 2: Add/update [pipenv] section in Pipfile

You need to disable pip input in Pipfile, this is the part that took me a while to figure out.

[pipenv]
disable_pip_input = false

Step 3: Install keyring and keyrings.google-artifactregistry-auth

In order for pipenv to know to use google authentication process you need to install the relevant keyring libraries.

pipenv install keyring keyrings.google-artifactregistry-auth --dev

Note if you have a package listed in the Pipfile that references the GCP source, then you will need to install the keyring libraries via pip first.

pipenv run python3 -m pip install keyring keyrings.google-artifactregistry-auth

Step 4: Add Google credentials

Either set the credentials with GOOGLE_APPLICATION_CREDENTIALS env var.

echo "GOOGLE_APPLICATION_CREDENTIALS=/path/to/gsa_key.json" > .env

Make sure to re-spawn the shell within the virtualenv to load the env vars from .env if necessary.

or you can make use of gcloud to login.

gcloud auth login

Refer to the Google Docs for more information on setting up credentials.

Step 5: Add package located in GCP Python artifact

You can now install any package from your GCP Python artifact.

pipenv install <package-name> --index=gcp

With the user of Traefik for the proxy, enable the Access Logs configuration and set the filePath option.

If you have multiple websites on the server and you want to track them separately (running their own instance of GoAccess) then you can setup a Systemd service that pipes the access logs to a separate log file for GoAccess to process for each of the websites.

For example;

/etc/systemd/system/multi-user.target.wants/example-analytics-filter.service

[Unit]
Description=Filter Example Access Logs

[Service]
Type=simple
ExecStart=/bin/sh -c '/usr/bin/tail -f /var/log/traefik/access.log | \
                  grep -i --line-buffered "\-example\-"'
KillMode=mixed
StandardOutput=file:/opt/goaccess/data/access_example.log
Restart=always
PrivateTmp=false

[Install]
WantedBy=multi-user.target

This service is continuously processing the Traefik access log file and updating the /opt/goaccess/data/access_example.log file. We will use this file as the log file for GoAccess to process.

Now create a GoAccess service file for each instance;

/etc/systemd/system/multi-user.target.wants/example-analytics.service

[Unit]
Description=Example Live Log Analyzer
After=example-analytics-filter.service

[Service]
Type=simple
ExecStart=/usr/local/bin/goaccess \
               --log-file /opt/goaccess/data/access_example.log \
               --log-format COMMON \
               --html-report-title "Example Statistics" \
               --port 7891 \
               --ws-url example.com:7890 \
               --real-time-html \
               --output /home/www/example/analytics/index.html \
               --geoip-database /opt/goaccess/geoip/GeoLite2-City.mmdb \
               --db-path /opt/goaccess/data/example \
               --persist \
               --restore
ExecStop=/bin/kill -9 ${MAINPID}
PrivateTmp=false
Restart=always

[Install]
WantedBy=multi-user.target

Seeing that we want to be flexible to have multiple GoAccess instances running for each site, we set the --port option to a different value for each instance, in this instance it is --port 7891, the next instance can be --port 7892 etc. For the --ws-url option we just need to change the domain name to match each website, but keep the same port number (7890) for each instance.

Lastly we have have the Traefik configuration to proxy everything. In the static configuration file we have the following;

/etc/traefik/traefik.toml

[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.web-secure]
    address = ":443"
  [entryPoints.analytics]
    address = ":7890"

[providers]
  [providers.file]
    directory = "/etc/traefik/conf.d"
    watch = true

[certificatesResolvers.letsencrypt.acme]
  email = "certs@example.com"
  storage = "acme.json"
  [certificatesResolvers.letsencrypt.acme.httpChallenge]
    entryPoint = "web"

[accessLog]
  filePath = "/var/log/traefik/access.log"

We have some middlewares we want to make use of;

/etc/traefik/conf.d/middleware.toml

http]
  [http.middlewares]
    [http.middlewares.https-redirect.redirectScheme]
      scheme = "https"
      permanent = true
      realm = "example.com"

   [http.middlewares.ssl-header.headers]
      [http.middlewares.ssl-header.headers.customRequestHeaders]
        X-Forwarded-Proto = "https"

Then for each website instance we have running;

/etc/traefik/conf.d/example.toml

[http]
  [http.routers]
    [http.routers.http-example-router]
      entryPoints = ["web"]
      service = "service-example"
      rule = "Host(`example.com`,`www.example.com`)"
      middlewares = ["https-redirect"]

    [http.routers.analytics-example-router]
      entryPoints = ["analytics"]
      service = "service-analytics-example"
      rule = "Host(`example.com`,`www.example.com`)"
      middlewares = ["ssl-header"]
      [http.routers.analytics-example-router.tls]
        certResolver = "letsencrypt"
        [[http.routers.analytics-example-router.tls.domains]]
          main = "example.com"
          sans = ["www.example.com"]

    [http.routers.https-example-router]
      entryPoints = ["web-secure"]
      service = "service-example"
      rule = "Host(`example.com`,`www.example.com`)"
      [http.routers.https-example-router.tls]
        certResolver = "letsencrypt"
        [[http.routers.https-example-router.tls.domains]]
          main = "example.com"
          sans = ["www.example.com"]

  [http.services]
    [http.services.service-example]
      [http.services.service-example.loadBalancer]
        [[http.services.service-example.loadBalancer.servers]]
          url = "http://127.0.0.1:3000"

    [http.services.service-analytics-example]
      [http.services.service-analytics-example.loadBalancer]
        [[http.services.service-analytics-example.loadBalancer.servers]]
          url = "http://127.0.0.1:7891"

What is not showng in this solution is the service that is running at http://127.0.0.1:3000 that is serving upthe relevant files for the example website.

In the example configurations above it is expected that the service running at http://127.0.0.1:3000 will serve the /home/www/example/analytics/index.html when prompted with the path /analytics.

Firstly you want to decide whether or not the Django application should be accessible directly by IP address, if so, make sure that the IP address is added to the ALLOWED_HOSTS setting.

If not, then you want to have the Django application listening on localhost and not the external IP address.

There are 2 ways for Traefik to handle checking the host headers;

Option 1:

The simplest way is to make use of the HostHeader rule in the router;

http:
  routers:
    router1:
      rule: "HostHeader(`example.com`, `www.example.com`)"
      service: myService

Option 2:

A more convoluted option is to make use of the Traefik allowedHosts middleware headers to allow only valid hosts.

A basic sample would be;

http:
  routers:
    router1:
      rule: "Host(`example.com`, `www.example.com`)"
      service: myService
      middlewares:
        - "myHosts"

  middlewares:
    myHosts:
      headers:
        allowedHosts:
          - "example.com"
          - "www.example.com"

And to ensure that it’s all working correctly;

# valid request
curl -H "HOST: example.com" http://example.com

# invalid HTTP_HOST header
curl -H "HOST: wrong.example.com" http://example.com

Over time the maintenance of the development environment generally requires updates to the services/servers and the adding/removing of users as the team members change. This could lead to a deficiency in the solution if not enough team members had the skillset or inclination to perform these tasks periodically.

So after having set up development environments a few times for different organizations. I started thinking it would be nice if there was a solution that provided this management of self-hosted environments, and hence Bayete was born.

I ended up down a rabbit hole, when attempting to check my SMTP settings in a Django application. An smtplib.SMTPServerDisconnected error was occurring each time. I thought this was related to closed SMPT ports (465/587), and went so far as to open a ticket with hosting provider asking for the relevant ports to be opened for my servers.

But alas, it was a simple bloody misconfigured username! So actually, I was failing to authenticate. Compounding the issue, was that locally I the correct username, but on the server I have the username misconfigured.

When I finally did each step manually, the issue was easy to see.

Below are some step by step actions to send SMTP email with python;

import smtplib
from email.mime.text import MIMEText

msg = MIMEText("Testing email")
msg['Subject'] = "Hello"
msg["From"] = "no-reply@example.com"
msg["To"] = "to@example.com"

s = smtplib.SMTP("smtp.example.com", 587)
s.login("postmaster@example.com", "supersecret")
s.sendmail(msg["From"], msg["To"], msg.as_string())
s.quit()

Can’t believe it took me so long, but now that I have, very happy with it. Nice and simple to make use of it.

I highly recommend taking a look at it.

On a reasonable size setup you can quickly hit that limit using JSON format, while YAML format will take you a while longer. A quick example of a file size comparison between the two file formats;

stack.yaml - 3.3K
stack.json - 5.6K

If you want to convert from JSON to YAML you can do that with the following python code snippet.

#!/env/bin python

import json
import yaml

with open("stack.json") as fj:
    with open("stack.yaml", "w") as fy:
        yaml.dump(json.load(fj), fy)

Another option is to handle large stack templates is to make use of Transform

I’m using TarFile package on python 3.5

My initial fixture was setup as;

@pytest.fixture
def file_tar(tmpdir, tmpdir_factory):
    fn = tmpdir_factory.mktemp("tar").join("test.tar")
    tf = tarfile.TarFile(str(fn), "w")
    p1 = tmpdir.mkdir("src1").join("test1.txt")
    p1.write("text 1 content")
    tf.add(p1)
    p2 = tmpdir.mkdir("src2").join("test2.txt")
    p2.write("text 2 content")
    tf.add(str(p2))
    tf.close()
    return fn

But I ended up with an empty file in the tar’d file fixture, and that was causing issues when attempting to extract the contents of the tar’d file. As it is an empty file and results in the following error/exception tarfile.ReadError: empty file

Changing the fixture to the following, resolved the issue;

@pytest.fixture
def file_tar(tmpdir, tmpdir_factory):
    fn = tmpdir_factory.mktemp("tar").join("test.tar")
    p1 = tmpdir.mkdir("src1").join("test1.txt")
    p1.write("text 1 content")
    p2 = tmpdir.mkdir("src2").join("test2.txt")
    p2.write("text 2 content")
    with tarfile.open(str(fn), "w") as tf:
        tf.add(p1)
        tf.add(str(p2))
    return fn

I guess that is more pythonic as well.

$ python setup.py sdist bdist_wheel upload


...


Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    install_requires=['django>=1.10']
  File "/usr/lib64/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib64/python3.5/distutils/command/upload.py", line 63, in run
    self.upload_file(command, pyversion, filename)
  File "/usr/lib64/python3.5/distutils/command/upload.py", line 162, in upload_file
    body.write(value)
TypeError: a bytes-like object is required, not 'str'

The setup.py file was along these lines

#!python
setup(
    name='project-name',
    ...
    platforms=['any'],
    license='MIT License',
    classifiers=(
        'Environment :: Web Environment',
        'Framework :: Django',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.5',
    ),
    packages=find_packages(),
    install_requires=['django>=1.10']
)

Took me a while to figure out what the issue was, and why the error, but alas is appears that the classifiers needs to be a list and not a tuple.

So changing the setup.py file to something like this

#!python
setup(
    name='project-name',
    ...
    platforms=['any'],
    license='MIT License',
    classifiers=[
        'Environment :: Web Environment',
        'Framework :: Django',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.5',
    ],
    packages=find_packages(),
    install_requires=['django>=1.10']
)

Fixed the issue.

This is a continuation of the Golang Web Apps and the source code can be found at https://github.com/reinbach/golang-webapp-guide.

To better organize the code we move all the templating functionality, the templates, and the static files into a separate directory. Which now gives us the following file layout.

.
├── server.go
└── template
    ├── constants.go
    ├── context.go
    ├── html
    │   └── ...
    ├── render.go
    └── static
        └── ...

The render functioned is tweaked in a couple of ways;

• A context parameter (data for the template) is passed to the render function
• A list of templates is now expected instead of a single template.

The UpdateTemplateList function is used to prepend the full path to the templates.

#!python
func UpdateTemplateList(tmpls []string) []string {
     d := GetAbsDir("template", TEMPLATE_DIR)
     for i, v := range tmpls {
         tmpls[i] = filepath.Join(d, v)
     }
     return tmpls
 }

And for that we made use of a constant, and so went a step further and moved all our constants into a sepatate constants file, which now has the following;

const (
      STATIC_URL     string = "/static/"
      STATIC_ROOT    string = "static/"
      TEMPLATE_DIR   string = "html/"
      PARENT_PACKAGE string = "template"
 )

If we wanted to rename the directories, we would need to just update this file.

With those above changes we now changed our home and about functions to the following;

func Home(c web.C, w http.ResponseWriter, r *http.Request) {
     ctx := template.NewContext()
     ctx.Add("HomePage", true)
     template.Render(c, w, r, append(templates, "home.html"), ctx)
}


func About(c web.C, w http.ResponseWriter, r *http.Request) {
     ctx := template.NewContext()
     ctx.Add("AboutPage", true)
     template.Render(c, w, r, append(templates, "about.html"), ctx)
}

Lastly we also make use of goji to handle routing and middleware extensibility for us. This requires a small changes to our main function in the server.go file. We simple swap out most of the http.HandleFunc calls with goji.Get and use goji.Serve() instead of the http.ListenAndServe(":8000", nil) So the main function looks like;

func main() {
    http.HandleFunc(template.STATIC_URL, template.StaticHandler)
    goji.Get("/", Home)
    goji.Get("/about", About)
    goji.NotFound(NotFound)


    goji.Serve()
}

As you may have noticed we also added a NotFound to catch any addresses that we cannot resolve.

func NotFound(c web.C, w http.ResponseWriter, r *http.Request) {
    template.Render(c, w, r, append(templates, "404.html"),
        template.NewContext())
}

And now we have a separate module that handles the template rendering and serving of static files for us.

Previously I have done a large amount of development with the Django framework. I like it because it has everything and it allows one to develop a website quickly, and so there will be a lot of influence from that in these articles.

The reason I made use of Golang is because I wanted to see how simple it would be to implement a lot of this functionality with mainly the Golang standard packages.

So following articles are available;

• Templating
• Testing
• Multiple Apps
• Config
• Form
• Database
• Logging
• Session
• Auth
• Cli

Each article attempts to cover a particular piece of functionality needed in a website. Source code is available on github at golang-webapp-guide

A while back I wrote about implementing basic site functionality, this will expand on that a lot more.

i.e. We don’t want all of the options returned by the foreign key to be shown.

I really like the Django class based views and use them as much as possible. I find that they make my life a lot easier. So when I needed to make use of some formsets in a particular view, but need the formsets to only display a subset of the elements referenced by the foreign key in the model, and tackled it in the following manner;

Here is our sample model, in this example when creating a transaction I only want to show those Accounts, referenced by account_debit / account_debit that are applicable / associated to the user making the request;

#!python
class Transaction(models.Model):
    account_debit = models.ForeignKey(Account, related_name="debit",
                                      verbose_name="debit")
    account_credit = models.ForeignKey(Account, related_name="credit",
                                       verbose_name="credit")
    amount = models.DecimalField(decimal_places=2, max_digits=8)
    summary = models.CharField(max_length=50)
    description = models.CharField(max_length=250, blank=True)
    date = models.DateField()

We create our normal model form, and the work happens in the TransactionBaseFormSet, we override BaseFormSet and in the _construct method we implement the logic that determines the account_debit/account_credit choices;

#!python
class TransactionForm(forms.ModelForm):
    class Meta:
        model = Transaction
        exclude = ["description"]




class TransactionBaseFormSet(BaseFormSet):
    def __init__(self, *args, **kwargs):
        self.user = kwargs.pop("user")
        super(TransactionBaseFormSet, self).__init__(*args, **kwargs)


    def _construct_form(self, i, **kwargs):
        form = super(TransactionBaseFormSet, self)._construct_form(i, **kwargs)
        account_choices = get_account_choices(self.user)
        form.fields["account_debit"].choices = account_choices
        form.fields["account_credit"].choices = account_choices
        form.fields["DELETE"].label = "Duplicate"
        return form




TransactionFormSet = formset_factory(TransactionForm, can_delete=True, extra=0,
                                     formset=TransactionBaseFormSet)

And to get the user object into the formset we create a class based view normally and in the get_form_kwargs method we pass in the relevant user object we need/want;

#!python
class TransactionImportConfirmView(FormView):
    template_name = "import.html"
    form_class = TransactionFormSet
    success_url = reverse_lazy("accounts.transaction.list")


    def get_form_kwargs(self):
        kwargs = super(TransactionImportConfirmView, self).get_form_kwargs()
        kwargs["user"] = self.request.user
        return kwargs

And there we have it, we just had to really override the BaseFormSet and weave what magic we wanted in the _construct method and ensure we pass in any parameters we needed with the logic as we’re all set.

The following is the way I went about solving this in this instance was to call django templates recursively.

Create a templatetag to handle the display of displaying indentation character the correct number of times;

#!python
from django import template
from django.template.defaultfilters import stringfilter


register = template.Library()




@register.filter
@stringfilter
def repeat(value, arg):
    return value * int(arg)

Then in the templates, we have the initial code that calls the rows template;

#!html
<tbody>
  {% for account in object_list %}
    {% with depth=0 %}
      {% include "row.html" %}
    {% endwith %}
  {% endfor %}
</tbody>

and the row template calls itself when applicable;

#!html
<tr>
  <td>
    {{ "-"|repeat:depth }}
    {{ account.name }}
  </td>
</tr>


{% if account.is_category %}
  {% with depth=depth|add:"1" %}
    {% for account in account.subaccounts %}
      {% include "row.html" %}
    {% endfor %}
  {% endwith %}
{% endif %}

####Caveats

• Not a ridiculous number of accounts, less than 100
• Do not expect a large number of levels, maybe 3 or 4 at most

I have not measured to see how crazy this may be in processing/resources as yet, but from everyday usage so far it has not been an issue. So if not experiencing any performance issues, I’m not going to waste time trying to find out if there are any.

The idea is that we start with the straight golang net/http package and slowly develop a more complex web application each time as our needs grow. Adding a component, or functionality to the system to solve the identified need.

All code can be found at https://github.com/reinbach/golang-webapp-guide

Basics

Firstly we start with the absolute basics. Which is to handle a request and return a response.

#!golang
package main


import (
    "io"
    "log"
    "net/http"
)


func Home(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "Hello World!")
}


func main() {
    http.HandleFunc("/", Home)
    err := http.ListenAndServe(":8000", nil)
    if err != nil {
    	log.Fatal("ListenAndServe: ", err)
    }
}

Save the above to a file (eg: server.go) and then you can run it with the following;

go run server.go

And pointing our browser at http://localhost:8000 you will get the following response;

Hello World!

Awesome, we have a web application! Sadly it only returns string responses, and we want more. We want to return HTML, so it looks like a proper website.

HTML

To convert to HTML responses we replace the “io” package with the “html/template” package. We also want more than 1 page in our website. So we create a function to render the HTML templates, to keep things a little DRY, update the Home function, and add an About page. Now our server.go file looks like the following.

#!golang
package main


import (
    "fmt"
    "html/template"
    "log"
    "net/http"
)


func Home(w http.ResponseWriter, req *http.Request) {
    render(w, "index.html")
}


func About(w http.ResponseWriter, req *http.Request) {
    render(w, "about.html")
}


func render(w http.ResponseWriter, tmpl string) {
    tmpl = fmt.Sprintf("templates/%s", tmpl)
    t, err := template.ParseFiles(tmpl)
    if err != nil {
    	log.Print("template parsing error: ", err)
    }
    err = t.Execute(w, "")
    if err != nil {
    	log.Print("template executing error: ", err)
    }
}


func main() {
    http.HandleFunc("/", Home)
    http.HandleFunc("/about/", About)
    err := http.ListenAndServe(":8000", nil)
    if err != nil {
    	log.Fatal("ListenAndServe: ", err)
    }
}

Don’t forget we need to add the index.html and about.html file. In the same directory as the server.go file create a templates directory and in that directory place the index.html and about.html files. So we have the following file layout.

.
├── server.go
└── templates
    ├── about.html
    └── index.html

So we have HTML pages, but they are lacking. We need to add some CSS to give our website some pop. Let’s make use of Bootstrap, it’s a fantastic base to build a website on. At the same time let’s make our templates a little DRYier by using a base.html file and the individual pages having just the content applicable to them.

STATIC

To serve our static content we can make use of http.ServeContent to do all the work for us. The nice thing about this, is that it handles the setting of the Content-Type for us. See the docs for more information.

#!golang
func StaticHandler(w http.ResponseWriter, req *http.Request) {
    static_file := req.URL.Path[len(STATIC_URL):]
    if len(static_file) != 0 {
    	f, err := http.Dir(STATIC_ROOT).Open(static_file)
    	if err == nil {
    		content := io.ReadSeeker(f)
    		http.ServeContent(w, req, static_file, time.Now(), content)
    		return
    	}
    }
    http.NotFound(w, req)
}

We also now have a base.html which is used in each response, so we tweak the render function to parse the base.html and pass in the Context to the template.

#!golang
func render(w http.ResponseWriter, tmpl string, context Context) {
    context.Static = STATIC_URL
    tmpl_list := []string{"templates/base.html",
    	fmt.Sprintf("templates/%s.html", tmpl)}
    t, err := template.ParseFiles(tmpl_list...)
    if err != nil {
    	log.Print("template parsing error: ", err)
    }
    err = t.Execute(w, context)
    if err != nil {
    	log.Print("template executing error: ", err)
    }
}

With those changes we now have the following for our server.go file;

#!golang
package main


import (
    "fmt"
    "html/template"
    "io"
    "log"
    "net/http"
    "time"
)


const STATIC_URL string = "/static/"
const STATIC_ROOT string = "static/"


type Context struct {
    Title  string
    Static string
}


func Home(w http.ResponseWriter, req *http.Request) {
    context := Context{Title: "Welcome!"}
    render(w, "index", context)
}


func About(w http.ResponseWriter, req *http.Request) {
    context := Context{Title: "About"}
    render(w, "about", context)
}


func render(w http.ResponseWriter, tmpl string, context Context) {
    context.Static = STATIC_URL
    tmpl_list := []string{"templates/base.html",
    	fmt.Sprintf("templates/%s.html", tmpl)}
    t, err := template.ParseFiles(tmpl_list...)
    if err != nil {
    	log.Print("template parsing error: ", err)
    }
    err = t.Execute(w, context)
    if err != nil {
    	log.Print("template executing error: ", err)
    }
}


func StaticHandler(w http.ResponseWriter, req *http.Request) {
    static_file := req.URL.Path[len(STATIC_URL):]
    if len(static_file) != 0 {
    	f, err := http.Dir(STATIC_ROOT).Open(static_file)
    	if err == nil {
    		content := io.ReadSeeker(f)
    		http.ServeContent(w, req, static_file, time.Now(), content)
    		return
    	}
    }
    http.NotFound(w, req)
}


func main() {
    http.HandleFunc("/", Home)
    http.HandleFunc("/about/", About)
    http.HandleFunc(STATIC_URL, StaticHandler)
    err := http.ListenAndServe(":8000", nil)
    if err != nil {
    	log.Fatal("ListenAndServe: ", err)
    }
}

And our index.html file looks like;

{{ define "content" }}
  <div class="page-header">
    <h1>{{ .Title }}</h1>
  </div>
  <p>We have a HTML web page, woot!</p>
{{ end }}

The about.html will look very similar, but with content changed.

And our base.html file looks like;

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Golang WebApp</title>
    <link rel="stylesheet" href="{{ .Static }}css/bootstrap.min.css" type="text/css">
    <link rel="shortcut icon" href="{{ .Static }}img/favicon.ico">
  </head>
  <body>
    <nav class="navbar navbar-default" role="navigation">
      <div class="container">
        <div class="navbar-header">
          <a class="navbar-brand" href="/">Golang WebApp</a>
        </div>
        <div class="collapse navbar-collapse">
          <ul class="nav navbar-nav">
            <li><a href="/about/">About</a></li>
          </ul>
        </div>
      </div>
    </nav>
    <div class="container">
      {{ template "content" . }}
    </div>
  </body>
</html>

We have set our static path as a constant in the server.go and are passing it to the base.html template via the context. This allows us to change the path/location of the static content in one location. Don’t forget to download Bootstrap and place the files in the static/ directory. Which should result in the following file layout;

.
├── server.go
├── static
│   ├── css
│   │   ├── bootstrap.min.css
│   │   └── ...
│   ├── fonts
│   │   └── ...
│   ├── img
│   │   └── favicon.ico
│   └── js
│       └── ...
└── templates
    ├── about.html
    ├── base.html
    └── index.html

Nice, now we have the basis for a decent website that can be styled to our heart’s content. At this point we have the basis for a static website.

Last week my commit message referenced the wrong ticket and in attempting to fix the commit message I accidentally reset our develop branch of the project to the master branch. In short, I wiped out all our changes in the project! A few months of work gone, poof! Well, Git is a distributed repo system so everyone still had the original correct repo, but to top things off I had also pushed these changes to our our central repo server. Awesome!

Anyway a quick chat with a team mate, who informed me about reflog in git, I was back to the races. A couple of deletes using reflog I was back to normal and breathing easy again.

git reflog delete master@{X}

With X being the reference number of the entry to remove.

So while git log will show you the current logs for the branch, git reflog records the updates on the branch tips.

So if you appear to have reset your branch to the beginning of time and you need to get back to the present. git reflog may be the tool for you to do that.

Previously I would go through the following flow;

• Get a notification that my server load was increasing
• Log into the sever
• Check/confirm that it was the usual issue, by looking at the logs
• Check where the offending IP address is originating from.
• Add an iptables rule to block the offending IP address iptables -I INPUT -s <OFFENDING_IP> -j DROP
• Watch as the load returned to normal

So after a little research, with the goal of not wanting anything fancy or crazy to install and/or setup. I decided to go with a few more iptable rules to try and mitigate the issue, as per Rainer Wichmann. It is an old posting, but looks exactly like the kind of rules I’m looking for;

iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set \
--name SSH -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -m recent --update --seconds 60 --hitcount 4 \
--rttl --name SSH -j LOG --log-prefix "SSH_brute_force "
iptables -A INPUT -p tcp --dport 22 -m recent --update --seconds 60 --hitcount 4 \
--rttl --name SSH -j DROP

There is also a whitelisting option, but seeing that I have a dynamic IP address, I didn’t see that as being exactly helpful.

Actually Hangman can be developed instead of a text adventure game if text adventure games are not your thing. As with a text adventure game, Hangman is very much straight programming, and nothing fancy.

The one interesting thing I did in my version of the Hangman game was to make use of the algorithm R(3.4.2) (Waterman’s “Reservoir Algorithm”) from Knuth’s “The Art of Computer Programming” (simplified version). I used it to generate my random word.

#!python
def get_random_word():
    # using algorithm R(3.4.2) (Waterman's "Reservoir Algorithm")
    # from Knuth's "The Art of Computer Programming" (simplified version)
    #
    # get random word from word file
    # - ignore words that are less than 3 chars
    with open(WORD_FILE, "r") as fp:
        line = next(fp)
        for num, aline in enumerate(fp):
            if len(aline) <= 2 or random.randrange(num + 2):
                continue
            line = aline
        return line.strip()

Source: StackOverflow

I grabbed a list of words from FreeBSD. This list of words is a text file with a word per line. So to select a random word I need to select a random line. With that in mind I used the above algorithm to randomly select a line from the word file, thus giving me my random word.

The rest of the program is pretty straight forward.

Source is at https://github.com/reinbach/hangman

Onto the next step…

So that is something I have done. The storyline is pretty much non-existant, but the programming part of it is there. I think, anyway. I actually enjoyed creating this more than I thought I would.

The nice part about it is that the framework is there. The storyling and flow of the adventure is added to the mapping.py and text.py files. So it is simple enough to develop the storyline into something reasonable. Definitely need to work on my storylines.

Source is at https://github.com/reinbach/text-adventure

Onto the next step…

So far the examples are pretty simple.

But I got to play with tickers and goroutines in go, so that was nice. I actually used the net/http package for handling the requests and did a little template parsing. It is really nice to be able to use base packages to achieve all this. It makes go really nice to play with.

The examples can be found at https://github.com/reinbach/sockjs-go-example

I’m sticking with the base packages for my first few apps, so I can actually learn them. When starting out with Python, I used Django mainly and I feel that was not the correct thing to do, as Django black boxed a lot of functionality and it took a while to get a deeper understanding.

Anyway back to golang and templating. Here I’m making use of the html/template package.

Create the base file;

#!html
<html>
  <body>
    <h1>{{ template "title" }}</h1>


    {{ template "content" }}
  </body>
</html>

So here we are expecting a couple of overrides, title and content which the templates that extend this base file need to provide. A sample file extending this base;

#!html
{{ define "title" }}About{{ end }}


{{ define "content" }}
  <p>About us page now</p>
{{ end }}

In this file you can see that we define the title and content information, which is used/needed by the base file.

Now we can parse these files with the html/template package and the final result will be the combination of these 2 files.

t, err := template.ParseFiles("base.html", "index.html")

The order of the files provided in this call is important. You will always want the base.html file to be the first file in the list, otherwise you will end up with a blank result.

A complete sample go file that does this;

#!golang
package main


import (
    "flag"
    "fmt"
    "html/template"
    "os"
)


var (
    file string
)


func init() {
    flag.StringVar(&file, "file", "index.html", "template file")
    flag.Parse()
}


func main() {
    t, err := template.ParseFiles("base.html", file)
    if err != nil {
    	fmt.Println("template parse error: ", err)
    	return
    }
    err = t.Execute(os.Stdout, "")
    if err != nil {
    	fmt.Println("template executing error: ", err)
    	return
    }
}

Make use the html files are in the same dir as the above file, you can then run it and see the results;

$ go run main.go


<html>
  <body>
    <h1>Home</h1>




  <p>We are at the home page</p>


  </body>
</html>

Or pass in the file option to use a the about.html file

$ go run main.go -file=about.html


<html>
  <body>
    <h1>About</h1>




  <p>About us page now</p>


  </body>
</html>

Take the above with a grain of salt. I’m just starting out with golang and this is just the solution I came up with that works well for me at the moment. Also note that html/template is an extension of text/template, so this should be applicable there as well, so not just limited to html.

Source files can be found at https://github.com/reinbach/html-template-example

So I started with Python, which is what I know best and created the following simple script;

#!/usr/bin/env python


def fac(n):
    if n == 0:
        return 1
    return n * fac(n - 1)


if __name__ == "__main__":
    t = 0
    for j in range(100000):
        for i in range(8):
            t += fac(i)
    print("total: {0}".format(t))

The reason for the total output, was to have a check to ensure that I was getting the same results in each of the scripts. To make sure that they are doing the same amount of work.

Running the script gives us the following execution time;

$ time ./factorial.py


total: 591400000


real    0m1.055s
user    0m1.053s
sys     0m0.000s

So I am getting about 1s in total execution time. Not bad.

Now the same code sample in C, to see what the time execution would be;

#!c
#include <stdio.h>


int fac(int);


int fac(int n) {
  if (n == 0) {
    return 1;
  }
  return n * fac(n - 1);
}


main() {
  int i, j;
  int t = 0;


  for (j = 0; j < 100000; j++) {
    for (i = 0; i <= 7; i++) {
      t += fac(i);
    }
  }


  printf("total: %d\n", t);
}

Compile and execute the above snippet of code;

$ gcc factorial.c -o factorial
$ time ./factorial


total: 591400000


real    0m0.029s
user    0m0.027s
sys     0m0.000

Ok, that’s quite an improvement. This is C, so we do expect there to be a great improvement.

Finally we create our code sample in Go;

#!go
package main


import "fmt"


func fact(n int) int {
    if n == 0 {
        return 1
    }
    return n * fact(n-1)
}


func main() {
    t := 0
    for j := 0; j < 100000; j++ {
        for i := range []int{1, 2, 3, 4, 5, 6, 7, 8} {
            t += fact(i)
        }
    }
    fmt.Println("total: ", t)
}

Then build and run the code sample and we get the following;

$ go build factorial.go
$ time ./factorial


total:  591400000


real    0m0.026s
user    0m0.020s
sys     0m0.003s

So, that’s pretty much the same as C, which is excellent. The best part of it all, is that it is actually fun to code in Go compared to C. Python was always an attraction for me as the language is a breeze to work with and enjoyable programming with it. Go is also a nice language to work with and it’s really fast to boot. So I’m very excited about the language.

The nice thing about Martini is that it relies heavily on net/http and it a very thin layer to it. Creating a web application is straight forward and simple enough. It will be interesting to work on a larger application with it to get a better feel for it.

These tutorials on it were interesting and helpful;

• Build a Christmas List with Martini
• Build a RESTful API with Martini

Started off with the Tour of Go, and lately the Go Advent has been very helpful. The Go Advent series was really nice as it introduced a wide varied list of applications/uses of go. A number of them that I would not have even looked at or been aware of.

I’ve just finished going through the By Example list, which was great as I find this method is the best way for me to learn. The next step is to work on a personal project and actually put together something using Go.

The question is what exactly should that project be.

You would think I would remember that I write these things. Anyway, it is fun seeing that there.

Well the fun was short lived, when I realized the link did not work. I had just updated my DNS for the domain and forgotten to setup the leading ‘www.’ for the domain… doh!

Blender is a tool that I have always wanted to learn how to use. I have been aware of Blender since the days when the community bought it from the company that went bankrupt and open sourced it. That was a number of years ago. And now I am actually able to do a few little things with it.

The only thing with the ‘Noob to Pro’ wikibook is that a number of the tutorials are for older versions of blender and I have some trouble converting the steps to the new UI.

I have now just realized the the User Manual is a lot more up to date with the latest version of Blender and it appears to be a better starting point in learning Blender. So now I am working through that, I’ve just gotten through the first few chapters/pages.

I have also been lurking in a number of forums on reddit. Mainly;

• Linux Gaming
• Game Dev
• Indie Gaming

And I am learning quite a bit from the postings there, from resources for programming to what to do to learn to draw.

I’ve been developing websites and web applications for over 10 years and I am looking to do something different. I still very much enjoy developing web applications and that is still what I do on a fulltime basis.

Now I want to develop games and do something a little differently, to learn new languages and create something other than web applications. I have always been interested in games, seeing that I play them a lot and my boys are growing up and playing them as well. So this would be something more to interact with them on as well. They can be my beta testers.

So this is going to be my little side project for a while. I expect it will take me probably about 5 years to develop a reasonable game, seeing that I have a lot to learn. The web applications I develop use Python mainly and sure you can develop games using Python with PyGame libs etc, but I want to make use of C/C++ or maybe Go (will need a lot od libs developed) as I want to learn more languages and expect the game I want to develop will need to make use of a faster language.

So it is safe to say that I am really clueless at the moment about game development, so this will be a long journey and that is ok. I’ll keep doing it so long as I am enjoying it. I’m not wanting to work for any game development company, I want to have a lot of freedom in deciding on what and how I do things. This is a hobby for me and I want it to be fun.

I think my first steps are;

• to initially go back and really learn C as a good basis.
• learn how to use Blender

My expectations are;

• to have fun
• learn new things
• this is going to be a long and slow process.
• doing something completely different to what I’m doing now
• maybe sell a game or 2

But after a few weeks of using Git extensively at WFP where we make use of the Git Flow process, I must say I am really enjoying Git and grasping the branching aspect of things a lot better. I would still not classify myself as a expert of Git by any stretch of the imagination, but I do feel like I have moved past the crawling stage.

As I’ve mentioned, we make use of Git Flow, so installing that helps things a little. It’s not really needed as the Git commands are simple enough, it does help with streamlining/automating a few tasks though.

See Git Flow installation instructions if you want to make use of it.

General

My daily Git routine generally revolves around the following commands;

# list of branches
git branch


# work on a particular branch
git checkout <branch>


# see status of changes
git status


# see changes for particular file
git diff <file>


# remove changes made to the file
git checkout <file>


# add a file to the list of files to be commited
git add <file>


# committing changes, -m if I want to add a comment inline with the command
git commit [-m]


# when merging, the Git commit message is pretty decent, so add to it, rather than overwriting it
git merge <branch>


# pull changes from remote to your branch
git pull <remote> <branch>


# push your changes to the remote server
git push <remote> <branch>


# delete remote branch, this command is slightly weird. Note colon, that's needed
git push <remote> :<remote_branch>


# delete a branch locally
# -D if there are changes in the branch you don't care about
git branch -d <branch>

See Git Documentation for more complete information.

Stashing

If I need to “park” my changes so I can do a pull/push or some action that needs the branch(es) to be ‘clean’, making use of stash is really helpful.

# show the list of currently stashed items
git stash list


# create a stash of the changes in branch
git stash


# apply the stashed item to the current branch
# this applies the stash to the current branch and leaves it in the stash list
# it applies the first stash item if none provided
git stash apply [<stash>]


# if you want to apply the first stash and remove it from the stash list
git stash pop


# to remove a stash item from the stash list
git stash drop [<stash>]


# to see the stash changes in greater detail
git stash show -p [<stash>]

See Git Tools - Stashing for more information on stashing.

Branching with Git Flow

Git Flow makes use of the following “types” of branches. Featrure/Release/Hotfix and each of those are handled in the same manner. So you just need to replace the command with the relevant type of branch you are dealing with.

# start a new branch
git flow <branch_type> start <name>


# to merge and remove the branch
git flow <branch_type> finish <name>

These are the commands I mainly use with Git Flow, there are a bunch of others, but I find the Git commands are simple enough to use. See Git Flow for more information of the other commands.

Undo

Oops, need to undo a commit

git reset HEAD~1

See Git-Reset for detail information on resetting.

My preferred Linux distro is Arch Linux, so when setting up the server on Rackspace that is the distro I select and the rest of the instructions are applicable to that distro.

Once the server has been created and is up and running, I update the system. See previous article on the steps I take to do that.

Setup PostgreSQL

pacman -S postgresql (Y)
systemd-tmpfiles --create postgresql.conf
mkdir /var/lib/postgres/data
chown -c -R postgres:postgres /var/lib/postgres
su - postgres
initdb -D '/var/lib/postgres/data'
systemctl start postgresql
systemctl enable postgresql


vi /var/lib/postgres/data/postgresql.conf
# set listen_address variable
set listen_addresses = '*'


# set password for postgres user
passwd postgres
su - postgres
psql -c "alter user postgres with password '<password>'" -d template1


# setup relevant connection parameters that you want to allow to have access
vi /var/lib/postgres/data/pg_hba.conf
# IPv4 local connections:
host   all   all   <your_desired_ip_address>/32   md5
# change the local users not be trusted by require a password
local   all         all    md5


# restart postgres to pick up changes
systemctl restart postgresql

Setup backup script

I have a simple database backup script that runs daily and makes a backup of each database.

# need steps to setup backup scripts
mkdir -p /opt/backup/database
chown -R postgres:postgres /opt/backup/database/


# move backup script into place
mv database_backup.py /etc/cron.daily/
chmod 700 /etc/cron.daily/database_backup.py


# update crontabs to run script daily
# add following to cron
MAILTO=<email_address>
0 3 * * * * root /etc/cron.daily/database_backup.py


# add .pgpass so script can access database
vi ~/.pgpass
*:*:*:postgres:<password>


# set decent permissions for file
chmod 600 ~/.pgpass


# install relevant database libraries so script actually works
pacman -S python-psycopg2

My preferred Linux distro is Arch Linux, so when setting up the server on Rackspace that is the distro I select and the rest of the instructions are applicable to that distro.

Once the server has been created and is up and running, I update the system. See previous article on the steps I take to do that.

Setup nginx

# install nginx
pacman -S nginx


# setup nginx to start when server starts
systemctl enable nginx
systemctl start nginx


# update nginx.conf (/etc/nginx/nginx.conf) file
vi /etc/nginx/nginx.conf


# comment out server settings
# and add the following to the bottom of the file;
include /opt/nginx/conf/sites/*;

I place the various sites nginx config files in /opt/nginx/conf/sites/ so need to create the directory.

mkdir -p /opt/nginx/conf/sites/

Setup uWSGI

# install uWSGI
cd /opt/
wget https://aur.archlinux.org/packages/uw/uwsgi/uwsgi.tar.gz
tar -zxvf uwsgi.tar.gz
cd /opt/uwsgi
makepkg --asroot
pacman -U uwsgi*.pkg.*
cp emperor.uwsgi.service  /etc/systemd/system/uwsgi.service


# setup uwsgi user and log files
useradd -c 'uwsgi user'  --system --no-create-home uwsgi
mkdir -p /var/log/uwsgi
chown -R uwsgi /var/log/uwsgi


# update emperor.ini file
vi /etc/uwsgi/emperor.ini


# emperor.ini contents
[uwsgi]
uid = <uwsgi_uid> # <- get this from the /etc/passwd file
logto = /var/log/uwsgi/uwsgi.log
emperor = /etc/uwsgi/apps
master = 1


# start up uwsgi
systemctl enable uwsgi
systemctl start uwsgi

Setup VirtualEnv

I setup each of the websites in it’s own virtual environment.

pacman -S python-virtualenvwrapper


# create/update ~/.bashrc and add relevant virtualenv wrapper details
vi ~/.bashrc


# add virtualenv wrapper details to .bashrc file
export WORKON_HOME=~/env
source /usr/bin/virtualenvwrapper.sh

Various other required libs

Install the relevant repo clients you make use of. For me that is Git and Mecurial.

pacman -S git

Depending on which databases you make use of you may need the relevant database libs. I make use of PostgreSQL for my database needs most of the time and the sites require psycopg2, which in turn needs the postgresql libs.

pacman -S  postgresql-libs

And that is it, I’m now all set to ran my fabric scripts, which setup and/or update the relevant site.

Thankfully things have gotten a lot simpler overtime and there are decent AUR packages that one can make use of. I’ve already converted these servers to be making use of systemd. See previous article on setting up a base server using systemd

On the remote servers do the following;

# required libs
pacman -S base-devel openssl


# Nagios NRPE
cd /opt/
wget  https://aur.archlinux.org/packages/na/nagios-nrpe/nagios-nrpe.tar.gz
tar -zxvf nagios-nrpe.tar.gz
cd nagios-nrpe
makepkg --asroot
pacman -U nagios-nrpe-*.pkg.tar.xz
systemctl enable nrpe
systemctl start nrpe


# Nagios Plugins
cd /opt/
wget https://aur.archlinux.org/packages/na/nagios-plugins/nagios-plugins.tar.gz
tar -zxvf nagios-plugins.tar.gz
cd nagios-plugins
pacman -S net-snmp
makepkg --asroot
pacman -U nagios-plugins-*.pkg.tar.xz

Lastly the NRPE config file needs to be tweaked, to allow requests from the Nagios/Icinga central server, update the location of the conf.d file, and to uncomment the commands you want to allow;

vi /etc/nrpe/nrpe.cfg


# set allowed_hosts to nagios server's IP (72.47.237.156)
allowed_hosts=<ip_address_of_nagios_server>


# update include_dir to absolute path for conf.d dir
include_dir=/etc/nrpe/conf.d/


# uncomment the commands you want to allow being called


# restart nrpe daemon to pick up changes
systemctl restart nrpe

NOTE: With this setup the NRPE is configured to not accept parameters on the requests. So you cannot make use of $ARG1$ etc in your commands. This is for security purposes. So you have to hardcode the relevant commands in the NRPE config file, or tweak the config file to allow parameters to be accepted.

You should now be all set to define the relevant services on the Nagios/Icinga server to check the remote server.

Once the server has been created and is up and running. These are the steps I go through to get the base system in place.

update system

pacman -Syu

convert to systemd

Currently Arch Linux still makes use of rc.d for running it’s daemons, but is in the process of changing to systemd. So we are going to move to systemd now, which should make things easier in the long run. Make use of the systemd installation instructions to do this;

pacman -S systemd

Edit your grub boot file and have the system boot into using systemd. Add the following snippet init=/usr/lib/systemd/systemd to the end of the kernel line, so it looks like kernel /boot/vmlinuz-linux root=/dev/xvda1 ro console=hvc0 init=/usr/lib/systemd/systemd. Then reboot the system.

vi /boot/grub/menu.lst
# update kernel line
reboot

After system has rebooted, you can confirm that systemd is running with the following;

car /proc/1/comm

Which should return the string systemd

Now we want to convert the relevant process to systemd;

systemctl enable sshd
systemctl enable syslog-ng

The final steps are to set the hostname and to clean up old packages and reset grub boot file;

hostnamectl set-hostname <your_hostname>
pacman -R sysvinit
pacman -S systemd-sysvcompat
vi /boot/grub/menu.lst
# reset kernel line
reboot

Once this is complete we now have a base system that we can save a server image of and then use this server image to create other servers from. We can now create Webservers or Database servers from this base server.

If you make use of Nagios/Icinga to monitor your servers, you can then setup NRPE on the remote servers.

The main change is with the uWSGI files and I am making use of the Emperor configuration rather. This is the suggested method if one has a number of apps on a single server.

This change required me tweaking the nginx config file and setting up a uwsgi configuration file for the websites. I modified the main nginx config file by adding include /opt/nginx/conf/sites/*; to it so each of the websites separate nginx conf files would be read and implemented.

Then for each website the following config files would be setup;

nginx config

server {
    listen 80;
    server_name *.example.com;
    error_log /var/log/nginx/example-error.log;
    access_log /var/log/nginx/example-access.log;


    location / {
        uwsgi_pass 127.0.0.1:9001;
        include uwsgi_params;
    }
}

This is a very simple example, the main part is the location section where I pass in the Port number that the uWSGI instance will be running on and the default include uwsgi_params.

uwsgi config

[uwsgi]
chdir = /opt/sites/example/
master = true
threads = 20
socket = 127.0.0.1:9001
callable = app
module = uwsgi_app
logto = /var/log/uwsgi/example.log
virtualenv = /opt/sites/example/
processes = 4

I make use of the ini config style and each website will have one of these. The socket value needs to match up with the uwsgi_pass value used in the nginx config file.

I make use of virtualenv to manage/handle the website instances and so the virtualenv option is included.

module points to the wsgi file in the website and that file will depend on the framework (Django, Flask etc. in my instance) being used for the website.

callable value needs to match the param that the wsgi file is using to set the website instance to.

Then depending on the framework used for the web site their would be a wsgi file setup to instantiate the website.

Django wsgi file sample

import sys
import os


sys.path.append(os.path.abspath("{0}/{1}".format(
    os.path.dirname(__file__),
    "example"
)))
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'


import django.core.handlers.wsgi


app = django.core.handlers.wsgi.WSGIHandler()

Note the app param is what needs to match what the callable value is in the uWSGI config file.

Flask wsgi file sample

import sys
import os


sys.path.append(os.path.dirname(os.path.abspath(__file__)))


from example import app


if __name__ == '__main__':
    app.run()

Granted I am not doing anything specifically that is having a profound effect on the world, or that I am not tackling some huge coding problem personally. But I believe the work I am doing now is a small contribution, and the sum of our parts in the organization is having a great impact on humanity and that makes me feel good.

WFP is a large organization and things move slowly compared to what I am used to. I am coming from a startup where things happened quickly. So that is going to be a bit of a transistion and something that I need to keep in mind.

On the opposite side of the coin, I think that WFP will have a much better life / work balance and that counts for a lot in my book.

As with everything there are pros and cons, but this is something that I have been wanting to do for some time now and I am very excited to be actually doing it.

The conference was in Santa Clara, CA this year, which is pretty much the epicenter of the start up world for programmers and software engineers. I had never been to the Santa Clara area before so it was nice to get a limited view of the area. Sadly I did not get to travel around the area much. I went to the conference, stayed at the hotel and headed home right afterwards.

It was impressive seeing the number of people there. There were 2,500 attendees. That does not seem like a large number of people, but seeing them all in one room/area makes you realize that it is a fair number of people.

There were people from all walks of life, a nice cross section. Something like 30 - 40 languages were spoken and ladies made up around 20% of the attendees. The whole event felt very relaxed with a common theme of how to be accepting and inviting to those wanting to learn Python or programming in general.

The talks were many and wide ranging. Thankfully most of them should end up on pyvideo.org for everyone to access them. As soon as I got home I was looking up one of the talks I attended on Speaker Deck, trying to get more information about it and to test things out. Some of the talks were packed, so it is a good idea to get to them early. Otherwise you could be standing or peering over someone’s shoulder.

There were also a number of scheduled periods for 5 min lightning talks, which doesn’t sound like much, but it was impressive. The topics were all over the place, with a lot of humour involved. This is something you want to make sure you get to see.

The question becomes, do you need to go to these conferences. It costs a fair bit and requires a large amount of your time. You can get all the information you need from the Internet, videos and slides of the talks at the conference are uploaded. These you can access from the comfort of your home. So why go? Well, even with all those options, I would say that it is absolutely worth your while in going.

It is very intensive, a lot of knowledge being presented on a wide range of topics. So you are being exposed and challenged on a number of fronts that you would probably not even consider. You get to see/meet people that you normally would never. Being an introvert, networking is something I am terrible at, but you end up having no choice. It’s happening whether you like it or not. And it really wasn’t so bad, because everyone I met was really friendly and helpful.

The next PyCon US is in Montreal, Canada. That’s a lot closer for me, and I will be attending that one, if I have my way.

Lastly, a big shout out to all the volunteers in making PyCon US 2013 happen. Thank you!

Lately I have been setting up a number of cPanel instances for a client and that is a pretty straightforward process. cPanel have decent installation instructions on how to do this and their script pretty much does most of the work.

One of the cPanel instances required to be PCI compliant and that wasn’t as clear compared to the installation instructions. The steps you will need to take to be PCI Compliant will depend on the company running the scan, the website you are scanning and what they identify as issues. In my instance it was SecurityMetrics who ran the scan and I ended up taking the following steps to become PCI Compliant;

CGI scipts, especially the guestbook.cgi script was a big no no. The quickest and most effective way to disable these scripts was to make use of an .htaccess file. This only works if you don’t want to make use of the guestbook.cgi script.

# /usr/local/cpanel/cgi-sys/.htaccess
RewriteEngine On
RewriteRule ^guestbook.cgi$ [G,L]

Packages with Security Issues. OpenSSL and BIND versions failed the CPI scan, but a quick check of the CVE values in the changelog of these packages indicated that the relevant issue had been resolved. is the CVE value that the PCI scan indicates that is failing. The step works for any package that is marked as having a security hole, if your system is already up to date.

rpm -q --changelog openssl | grep <CVE-string>
rpm -q --changelog bind | grep <CVE-string>

If the results show in the changelog that the CVE value has been fixed you can then inform the entity doing the PCI scan.

Apache Configuration. In the cPanel interface under Sevice Configuration > Apache Configuration > Global Configuration. There are PCI Recommendations for some of the settings. Follow these recommendations. Don’t forget to rebuild the apache configuration and restart it, otherwise the changes will not take effect.

Firewall. Finally install ConfigServer Security & Firewall (CSF). The installation is pretty straightforward and once that is done, you can config and run tests via the cPanel interface. This resolved most of the issues, once I had installed it and followed most of the suggestions provided when running it’s test.

Resources

• CGI Script
• CVE Checks
• PCI Compliance in cPanel
• Moving Towards PCI Compliance with cPanel
• CSF

We are very much a Django shop at the company I work at and have moved to a Service Orientated Architecture (SOA). Well there are some legacy applications that are in the process of being moved into this sort of artchiture. A number of libraries have been used to help with the various APIs developed, these have mainly been Piston and TastyPie.

There are great discussions and comparisons out on the Tubes about these various libraries. But I must say version 2 of Django REST Framework is looking great.

Django REST Framework feels like it handles a lot more possibilities, that you are not limited or forced down a particular path. You have the Serialization shim, but other than that you can structure your WEB Api code the way that most suites you.

You can make use of functional or class views, but if you have a general model and setup and no need to get fancy there are the relevant helper classes that do that necessary magic for you. Actually it appears there are helper functions/classes at all levels no matter which direction you can with structuring your WEB Api. There are even mixins you can make use of if you make use of classes for your views and want a little help with some functionality.

Their tutorial is great and covers all these areas. The added bonus of this library is the Web Interface that pops up, it really helps with creating data and interacting with the WEB Api a lot simpler.

Well that escalated quickly! I was wanting to update the look and feel of this blog, felt it needed a little touch up.

I had read and heard about various blog systems that were very static, with the relevant files being pre-generated and pushed to the server. The benefit being that this simplified things on the server side, server would be able to handle large loads etc. Also the blog entries could be versioned stored in a repo easily enough.

So it appears things lined up this past week. I ran across a blurb saying that kernel.org just updated their site to make use of Pelican, and I decided to take a look at it. Well that quickly turned into me trying out the tutorial, then moving onto the rest of the documentation and working on developing a theme and before I knew it, I had just redone my site.

I created a quick python script to pull my blog entries and generate the relevant markdown files. Luckily my blog makes use of markdown, so that was simple enough.

A quick nginx conf change, which included adding a rewrite rule for old links and voila things are up and running with a new theme etc.

So I’m extremely impressed with Pelican, they have done a fantastic job and it’s very easy to use. The packaged commands with it make it a breeze doing the various tasks you need to do. I really like making use of my normal editor (Emacs) to write articles and the bonus is that the articles are now stored in the repo as well.

Generic view changes

This has changed a bit, and I was only using it to generate the robots.txt file. So I just moved it to a simple view. So no longer use generic views.

Syndication contrib changes

I just needed to change the import path and tweak the url settings file a bit. Actually a lot simpler now. See the syndication page for more information.

ALLOWED_HOSTS

Due to potential CSRF issues this setting is now needed in the settings.

ALLOWED_HOSTS = ['.reinbach.com']

The dot allows any subdomain to work as well.

Admin static files

The path location changed, it is now. So I just needed to change my nginx.conf file to have the /static/admin path point to the new location, which is;

django/contrib/admin/static/admin/

That was pretty much all I needed to do, now I’m at the latest and greatest. Granted this site is really basic with not much functionality within it. So no major tweaks needed. But still nice when it is quick and simple to jump up to a new release.

Well that was added simply enough on the client side, but it took me a while to realize why it was not working on the server side of things.

I finally realized that when adding new headers to the requests, I need to add them to the headers list in the crossdomain decorator so that they pass through;

@app.route("/login", methods=['POST')
@crossdomain(origin='*', headers='origin, x-requested-with, content-type, accept, authtoken')
def login():
    ...<snip>...

Otherwise it would be stripped out of the headers and I would end up constantly asking the client to authenticate and get no where.

It’s very handy to be able to write a snippet of Javascript code in your default editor and then use node to run it. Especially for learning and just trying something out. I find that a lot better than trying to make use of the browser Javscript consoles or scratch pads.

Also Node is really fantastic when making use of testacular etc for running the various unit and end to end tests on ones project.

I’m not mkaing use of Node just yet in production as I’m happy with my current production setup (nginx, uwsgi) and would rather continue to keep that simple and minimal for now.

For the API, I used the following view decorator and I also added

f.required_methods = ['OPTIONS']

to the decorator function as suggested in the comments. I then implemented the decorator in the following manner;

@app.route("/login", methods=['POST')
@crossdomain(origin='*', headers='origin, x-requested-with, content-type, accept')
def login():
    ...<snip>...

Once that was done the calls from the JS Applicatin worked.

To improve things, I’ll probably move the headers param into the decorator function and make it a default rather than having to type it every time.

So I decided to make use of Werkzeug and ended up with the following straight forward python code to deliver the application code.

I wanted to stick with the angular-seed application format/layout of code and so the static files are delivered via the SharedDataMiddleware with the relevant paths defined there.

HTML files are delivered in the normal manner. A couple of improvements could be to only allow .html files and robots.txt files. It’s still early days in the development of this project, so things may change as I learn more.

This week I wanted to get back to working on JS frameworks/application as I need one for my Finance App. For the browser UI application portion of the app, I wanted to make a purely JS application and I decided to try out AngularJS after reading some good things about it.

I’ve since gone through their Tutorial and was introduced to Testacular for running unit tests and Jasmine for the end 2 end testing. In the past I’ve not done much in the way of testing for JS code and these look fantastic, and hopefully I will get a lot better with testing JS code. I plan to do a lot for this application.

At the moment, I’ve cloned the angular-seed project and starting to setup my application, while I work my way through the Guide. The examples and the testing make use of node.js, which is fine for unit testing. But my production servers make use of nginx and uWSGI for the webserver, so I’m needing to sort out that aspect of things before progressing too much.

So this requires me to allocate each of the expenses to the relevant expense account, as I need to draw the relevant amount from each of the expense accounts.

I actually use GnuCash to track my finances, which is a great application and has served me well. The issue I have is with my process and I am starting to find it a bit montonous having to work out the various expenses on the credit card and allocate them accordingly. So I’ve finally decided to do something about it and put together an application that will hopefully make things a lot more steamlined for me.

For some time now I have been tending towards developing applications that have a clear and distinct separation between the UI aspect of the application and the business logic. In order to accomplish this I am developing a solid API application that can serve/handle the various UI applications.

The various UI applications are applications that are developed for the browser, Android, IOS etc.

I’m still working through the various nuances of making this work the way I like and hopefully by the end of this application I will have a solid framework/method of how best to handle this.

To see how things are going you can follow via the GitHub repo

• develop Finance app and at least 5 other mini apps
• Learn 2 new languages (Go, Haskell)
• Develop at least 1 App for Android and IOS
• Read the following books; The C Programming Language, Programming Pearls, The Algorithm Design Manual
• Improve my writing: write 52 blog entries (1/week)
• Go on 3 family vacations
• Excerise on a regular basis

First things first get XCode and install the Command Line Tools package, and make sure it is up to date. Or you can make use of the stand alone CLI package.

Next install “homebrew”, can’t leave home without it.

With most of my development making use of Python, I make use of virtualenv and virtualenvwrapper to separate the various projects.

brew install virtualenv
brew install virtualenvwrapper

I like to make use of Gevent in a lot of projects, but to be able to installed gevent in the various virtualenv, libevent needs to be installed globally for gevent to build against.

brew install libevent

MySQL is used as the database for some projects. Grabbed the and ran into an issue with the lib not being found, so created a symbolic link for it;

sudo ln -s /usr/local/mysql/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib

Less is used for css styling and that is compressed as well. So needed node.js and relevant less package;

brew install node.js
curl https://npmjs.org/install.sh | sh
npm install --global less

This pretty much gets me to the state that I like to have as a development machine. Hopefully I remembered everything here.

I wanted something that was easy to add news tests to and allow me to spread them over multiple files/modules. The project has the following dir. structure;

|- config.py
|- /project
    ... snip ...
|- /tests
    |- api_config_type.py
|- runserver.py
| -test_runner.py <- this is the test runner script

So the script starts off updating the sys.path, as to allow the tests to be able to access the relevant modules in the project

sys.path.append(os.path.abspath(__file__))

Next I setup the list of the test modules I want to pull in a run each time I call the test runner. This is a simple list of them;

test_modules = [
    'tests.api_config_type'
]

The next bit is the meat of it all. Here we loop through the test_modules list and import the test_cases from them which is another simple list. The test cases list in the module would look something like this;

test_cases = [APIConfigTypeTest, APIConfigKeyTest]

Which holds the test cases you want to include in the test runner. So as the test runner loops through these test cases it uses UnitTest’s test loader to load the test cases

suites = []
for test_mod in test_modules:
    _temp = __import__(test_mod, globals(), locals(), ['test_cases'], -1)
    for test_case in _temp.test_cases:
        suites.append(unittest.TestLoader().loadTestsFromTestCase(test_case))

The result of this is a master suites list of test cases to run. We add that to the test suite and then call the test runner to run them;

alltests = unittest.TestSuite(suites)
runner = unittest.TextTestRunner()

That’s it, not perfect could do with some improvements, but it gets the job done and I’m back to writing more tests and adding them simply enough to the test runner.

The complete script (test_runner.py);

import os
import sys
import unittest


sys.path.append(os.path.abspath(__file__))


test_modules = [
    'tests.api_config_type'
]


suites = []
for test_mod in test_modules:
    _temp = __import__(test_mod, globals(), locals(), ['test_cases'], -1)
    for test_case in _temp.test_cases:
        suites.append(unittest.TestLoader().loadTestsFromTestCase(test_case))


alltests = unittest.TestSuite(suites)
runner = unittest.TextTestRunner()


if __name__ == "__main__":
    runner.run(alltests)

https://github.com/reinbach/mini-triage

The first issue was simple enough and uwsgi logs provided the needed information. Python imports were not finding the needed packages. I tweaked the code to update the python paths.

The second issue took me a moment to work out. I kept seeing the following error in the logs;

"application" must be a callable object in file....

Now I know I was telling uwsgi via nginx conf file that the application callable object was “app”. But I got stuck on thinking that nginx was expecting the callable object to be “application” so I changed the app around to use application, but obviously that did not work.

A little research showed that web.py needs to make use of the following for uwsgi to work;

application = app.wsgifunc()

Tweaked that to be;

app = app.wsgifunc()

And everything worked like a charm. The following is the wrapper I used on the application to handle the above issues.

import sys
import os


prev_sys_path = list(sys.path)
sys.path.append(os.path.dirname(os.path.abspath(__file__)))


new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
    sys.path.remove(item)
    sys.path[:0] = new_sys_path

from code import app


app = app.wsgifunc()


if __name__ == '__main__':
    app.run()

The part “from code import app” is the part that pulls in the web.py application. This file is set as the app and as callable in nginx conf file.

Now with “real time” apps or apps that have a more “desktop” like experience you want to be improve the connection/speed between the client/browser and the server, so keeping a connection open and streaming the relevant data to the browser really helps with this. It’s a lot better than having to refresh the page continuously and provides the impression that the web app is more responsive.

This is where socket.io comes into play. It handles most of the issues with keeping this connection open, and especially handles the nuances of each browser. It will make use of websockets where possible and gracefully fail down to using flash for those older browsers.

Now seeing that I make use of Python for more programming fix, thankfully there is the nice package for this called gevent-socketio that makes things simpler.

So with that I have created a few simple examples with it to get a feel for using it;

• gevent-socketio-example
• gevent-zmq-socketio-example

Really nice and this opens a lot of possibilities which I hope to play with now.

So I created a mini app that pushes updates/additions to data in the browser to all users viewing that data.

https://github.com/reinbach/datashare

In this mini app the data in the browser is a table with each cell being editable and with the ability to add a new line to the table. As changes are made in each of these cells or a row is added, any of the users viewing this data will see these changes in their browser instantly.

The servers had a normal apache instance running, without nginx etc. So I made use of .htaccess and a html page to handle this.

I put in place a maintenance.html file on the server that looked like this;

<html>
<head>
    <title>Pearl Izumi</title>
    <script type="text/javascript">
    function delayer() {
        window.location = "/";
    }
    </script>
</head>
<body style="text-align: center;" onLoad="setTimeout('delayer()', 5000)">
    <h3>Currently in maintenance mode</h3>


    <p>We're making this a better place.</p>
</body>
</html>

The javascript is in there to redirect the use back to the home page every 5s to help them keep checking, also to redirect them away from the maintenance.html page. Otherwise when we updated the .htaccess file, and the user tries to reload the site, they may still end up at the maintenance page. Also this does not work well if a user has javascript turned off. So there are a number of things that need to be tweaked to improve this.

The .htaccess file redirects all requests to the maintenance page and is done with the following;

RewriteEngine On
RewriteCond %{REQUEST_URI} !/maintenance.html$
RewriteCond %{REMOTE_HOST} !^67\.255\.207\.160
RewriteRule $ /maintenance.html [R=307,L]

If you need to open the site up to other IP addresses, you can add extra “REMOTE_HOST” lines with the relevant IP address. In this instance all requests are sent to the maintenance.html page, so that makes it tricky to add images and styling to your maintenance page. To do that in this case, you would need to link to images/css files off this server at another server.

So that was it. To turn the maintenance page on and off, just un/comment the rewrite rules in the .htaccess page. You probably also want to rename the maintenance page to something else, that is not accessible by the webserver, to prevent people accidentally accessing it.

I now also make use of this maintenance page when doing updates to the site.

So I created a python script to do the work for me, it will handle any new databases that are added, so I don’t have to worry about setting up a backup script each time I create a new database.

You can find the script at https://gist.github.com/90b45a8259ad0b906554

The script makes use of the .pgpass file, so you need to setup a .pgpass file for the user running the script, that stores the access params to psql so that the pg_dump call will work. See [see http://www.postgresql.org/docs/8.4/static/libpq-pgpass.html](see http://www.postgresql.org/docs/8.4/static/libpq-pgpass.html)

The script is something to get started with and will only work with small databases that have low traffic.

Setup up a crontab to run the script.

0 3 * * * * root /etc/cron.daily/psql_backup.py

Then there is a separate rsync script to pull the database instances off the server to another location/data center.

I followed the quickstart guide.

I have a server outside of rackspace and decided to make use of that to monitor the rackspace servers, sadly though this server runs CentOS and did not have the latest versions of PCRE (> 7.6) version available in order to run the new web interface available for Icinga, so had to resort back to the classic interface.

Seeing that I was monitoring a couple of remote servers and hopefully the number of them will grow over time, I decided to make use of Nagios NRPE plugin for the remote monitoring. This is meant to be a less resource intensive manner in monitoring although a little less secure than using SSH.

This required setting up the Nagios nrpe plugin on the remote servers as well as the Nagios plugins. I made use of both the nagios setup and the Icinga NRPE instructions for the remote servers. The extra step was to create a user/group for nagios, otherwise the make install would not work.

When making changes to your configuration, it’s a good idea to run one of the following before reloading or restarting the Icinga daemon;

/usr/local/icinga/bin/icinga -v /usr/local/icinga/etc/icinga.cfg
/etc/init.d/icinga checkconfig
/etc/init.d/icinga show-errors

After making sure protobuf was installed system wide, currently the latest version is 2.4.1. In my virtualenv I did the following to get it all sorted out;

wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2
tar -jxvf protobuf-2.4.1.tar.bz2
cd protobuf-2.4.1/python/
python setup.py install
pip install riak

Make sure you download the protobuf version that matches what you installed system wide. See download page

Running “pip install riak” first and it was attempting to pull in a protobuf library from basho and for some reason it was failing. So did the above and things look good.

Today I tried out BrowserStack. I had to debug a site in IE9, and after a quick signup, I was at the dashboard and able to load up the site in IE9 and troubleshoot my issue. I was given 60min of free time, which is nice, as it allowed me to try out the service and get comfortable with it.

The neat thing about BrowserStack is that they provide a number of debug tools, eg: Firebug Lite, in the browser that they start up. You don’t have to install these, all you have to worry about is the issue you are dealing with.

A couple of caveats came up;

• I tried to do Local Testing and that crashed on me. It makes use of a Java applet and that is what appeared to crash on me. I was running Chrome for this and did not try other browsers, as I just moved to a URL that was accessible outside of my dev environment. A troubleshooting ticket was automatically sent to them, so hopefully that will get sorted out.

• There is a little latency when using the service, but with what is actually happening that is expected. It was not horrendous and I was still able to be productive. Not something I would like to spend a lot of time in, but fine for troubleshooting and quick testing.

So overall I’m happy with the service, makes me excited to think that I can blow away my VM Window instances and no longer have to worry about dual booting etc.

[update 2/23/2012] After using this service some more, I’ve run into issues with it being very slow and painful to use at times. I was not as productive as it was initially. So I’m heading back to the good ‘ol VMs.

So the idea is that I want to really have a single source for the documentation, and ideally it would be driven by the code. But alas that is really hard to do, so the idea is to add docstrings to your code. These can be read by the sphinx code during a build to help generate the necessary documentation. The markup/down makes use of reStructured Text which is simple to learn and make use of.

I ran a test build against a project I had just started and was very impressed with the results. I actually got a fair amount of information from my code that I told the sphinx build to include. The code had very little dostrings in it at the time, so that was nice to see.

There is a sphinx-quickstart script that helps to generate the conf.py file and there are a large number of available settings/options you can choose. One being the creation of a makefile, which makes it very easy to rebuild the documentation, with

make html

And you have HTML files created of your documentation. You can switch out the “html” option/argument for Latex and various others.

This is just a scratch of what Sphinx is capable of, but I have the basics down and will continue using it and growing/developing my documentation of my codes and projects.

There is a very cool documentation hosting tool out there called Read the Docs and that can import your Sphinx generated docs.

My first impressions is that this is very sweet. I am very used to the ORM in Django and have hit a number of limitations with the way it works. I have not enjoyed having to jump Q object to start making use of AND and OR statements. Also the Aggregate and Annotate options are a bit quirky. The Django ORM does the basics and it does it well. But as you need to do more and more complex queries you start to run into limitations, and things start to feel like there is a fair amount of patching.

I’ve just finished the guide on SQLAlchemy and it appears to provide a very well rounded database abstraction layer. So am very keen to get using it in my project.

Obviously in both options you can always resort to raw sql, but then that kinda defeats the purpose of making use of an ORM. And yes you can insert SQLAlchemy into Django if you like, but I already prefer Jinja2 and pulling up the Django ORM, then there is a lot of patching happening on a framework from the get go and that is not always a good thing.

So I created a simple web app called DipToe that accepts and stores email addresses in a flat file. Made use of the Flask framework and in about 70 lines of code, excluding the html and css, have the site complete. It prompts for an email address and stores this in a flat text file.

There is validation on the email address, which I pulled out of the Django framework. It also very loosely checks whether the user has entered the email address already and doesn’t save it again.

The layout of the page has a large “splash” area on the home page and an about page as well. It is easy enough to add more pages if needed.

I made use of the twitter/bootstrap for most of the css.

• Jinja2
• Django
• MySQL / Postgres
• nginx
• Apache

And based on some of the tasks I have been needing to do (queueing) and handling huge load issues that I have started to encounter. I have started to re-look at this stack.

I definitely need to get some decent queueing mechanism in place, we have started to make use of Celery at work. But ZeroMQ sounds really interesting to me, so want to have a look at that some more. Would like to try it out on some projects soon. So some queueing/messaging layer needs to be added to the stack.

I just read the great article on benchmarking python wsgi servers by Nicholas Piël. Based on the results of all the benchmarking he did and what I am seeing/reading about what others are using in their stack I think I am going to try out gevent and uwsgi.

I have also started seeing people making use of different types of databases based on the specific need. This varying from graph databases like neo4j to key-value store like redis to document orientated databases like mongoDB.

For the last few years I have been using Django for my framework and this works extremely well. As you get everything that you would need when developing full featured web applications. There is a great community around it and a huge amount of information out there. But since I started doing some Google AppEngine work, Django did not work very well for that, it is not engineered to and there are better frameworks to do this (since then django-nosql has been developed). It got me thinking about other python frameworks and the possibility that they are better in certain instances.

So I have been playing with a few of those and expect that some of them will become part of the stack based on the need. In a similar way that various database instances/types are applicable in certain instances. The main ones that I have looked at are webapp2, webpy and Flask

On the client side of things, HTML5 is gathering a lot of steam and a number of interesting projects are cropping up around Javascript and CSS. Well when I say cropping up, it is relative to what I am only starting to learn about. May have been around for a while, but I am only just now starting to find out about them.

As the use of javascript has grown, fairly large applications have started to be built in javascript and so some straight Javascript frameworks have been created like Knockout, SproutCore, and backbone.js. There is also CoffeScript which is a new language that compiles into Javascript.

While for CSS you have pre-processors being developed that help with the creation and maintenance of stylesheets like Less, Sass and OO Css

Lastly an interesting area that growing in popularity is opening a connection between the client (talking the browser here) and server allowing messages to be sent along this connection. Websockets and socket.io appear to be the most interesting in this area at this point.

So loads of new cool technologies to wrap my head around and learn about. Hopefully implement them in a project soon to get a real feel about them. My stack is starting to expand and that is a great feeling as it makes it a lot more robust and scalable. I’m also able to handle and apply better and proper tools to the required solution/application.

Developing on a mac I found that installing GPGTools was the easiest route in doing that. On the servers we would do the normal compiling of the relevant source code to install. There are currently 2 versions of GPG out there 1.41 and 2.018. GPGTools installs both and a bunch of other goodies.

To install python-gnupg, the ever faithful pip does the trick

pip install python-gnupg

Now you can do everything you need to do through python. If you run into any issues, a good resource is either the docs or to actually go and download the source and there is a file there called test_gnupg.py that gives you all the example code you need.

So if you have issues or start getting 505 errors. It’s due to memory issues!

[updated] It appears there was a conflict in the apache config files dealing with SSL certs. Able to remove these conflicts and things to be a lot stabler of late. The interesting thing is that this is a VPS Server and I am meant to be limited to 512MB or so, but if I run

free -m
                 total       used       free     shared    buffers     cached
Mem:           894        416        478          0          0          0
-/+ buffers/cache:        416        478
Swap:            0          0          0

It was showing a total of 894 MB and that I still had 150MB free, and whenever it got to that point services were being stopped. It appears that PostgreSQL was the first service to go, meaning this site would go down. So that is something to change in regards to monitoring.

There was plenty of resources out there in setting up a virtualenv, so it is assumed that you have that as well as having virtualenvwrapper and pip installed.

Create VirtualEnv

mkvirtualenv --no-site-packages sample

uWSGI

Install uwsgi making use of pip, makes life nice and easy;

pip install http://projects.unbit.it/downloads/uwsgi-latest.tar.gz

nginx

Now we need to break out our compiling chops to install nginx. You can find the latest version on the download page I downloaded version 1.1.2 and untar’d it all through the browser. Next thing is in the terminal head on over to the untar’d dir

cd ~/Downloads/nginx-1.1.2
./configure --prefix=/usr/local/ --with-http_ssl_module
make
sudo make install

I added the –with-http_ssl_module config option as I am expecting to do some SSL work on my projects. This sample does not make use of it at all. You no longer have to add the –add-module=../uwsgi/nginx/ config option as nginx automatically includes that now.

Next you need to configure the nginx config files. I made use of the sites-available sites-enabled dir structure to manage the various configured local sites.

cd /usr/local
make nginx
cd  nginx
mkdir sites-available sites-enabled
touch sites-available/sample
ln -s sites-available sites-enabled/sample

We need to tell nginx to make use of the config files placed in these directories and we do that by adding the following line;

include /usr/local/nginx/sites-enabled/*;

Right before the last } in the /usr/local/conf/nginx.conf file.

Now we can add the various nginx config files per website/project. You can use the following template as a starting point. It should get you up and running at least. Add this to the sites-available/sample file.

server {
    listen 5001;
    server_name localhost;
    set $home /virtualenv/sample/hello;
    access_log /virtualenv/sample/hello/logs/nginx/access.log;
    error_log /virtualenv/sample/hello/logs/nginx/error.log;

    rewrite ^/(.*)/favicon.ico$ /static/images/favicon.ico last;

    location / {
        uwsgi_pass localhost:3031;
        include uwsgi_params;
        uwsgi_param USWGI_CHDIR $home/deploy;
        uwsgi_param USWGI_INI deploy;
        root $home;
    }
    location /static/ {
        root $home;
        autoindex on;
        error_page 404 = "404";
    }
}

The “hello” part is our expected sample flask project we are going to create. And “deploy” refers to dir/files we are going to create in the “hello” project. I also added the rewrite rule as that helps prevent the logging of 404 attempts on the favicon which is always asked for by modern browsers nowadays.

Flask

Head back to your virtulenv and keep the love going with pip to install Flask;

workon sample
pip install flask

We now create the hello world sample project

mkdir hello
cd hello
touch deploy/deploy.ini

Now edit the deploy.ini file and add the following to it.

[uwsgi]
socket = 127.0.0.1:3031
processes = 2
virtualenv = /virtualenv/sample/
module = myapp
callable = app
touch-reload = /virtualenv/sample/hello/

The touch-reload option is nice for dev, but not a good idea for production systems. In that anytime a file is touched with in the hello dir uwsgi will reload the changes. Saves you having to restart the uwsgi service. The “myapp” module and “app” callable relates to the app we are yet to create.

Ok finally to the creating of the sample app. Create a myapp.py file in the hello dir and add the simple hello world flask example;

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return "Hello World!"

if __name__ == '__main__':
    app.run()

We should now be all set to start up the processes and test it out.

uwsgi --ini deploy/deploy.ini
/usr/local/sbin/nginx

Fire up the browser and point it to http://localhost:5001. Hopefully you get the “hello world!” message in your browser.

If you need to stop nginx or reload changes you can do the following;

/usr/local/sbin/nginx -s reload
/usr/local/sbin/nginx -s stop

I’ve been needing to get some decent messaging layer in place and so I decided to try zeromq out.

Grabbed the latest version (currently 2.1.9)

cd /path/to/downloaded/zeromq-2.1.9
./configure --prefix=/usr/local
make
sudo make install

Then because I make use of virtualenv, I wanted to install the pyzmq libs in the specific virtualenv I was working on. First I downloaded the pyzmq lib (currently 2.1.9

source /path/to/virtualenv/bin/activate
cd /path/to/downloaded/pyzmq-2.1.9
python setup.py  configure --zmq=/usr/local
python setup.py install

And just to confirm that all is well, you can do the following;

python -c "import zmq"

If you get no errors you’re golden.

Now onto the ZeroMQ Guide. It is a pretty lengthy endeavor, but there are loads of great sample code in many different languages. The guide and sample code, can also be accessed from GitHub.

./configure --prefix=/usr/local
make
sudo make install

There after you can use pip to finish up installing the rest of the requirements and gevent itself

pip install greenlet
pip install gevent

A quick test to make sure you’re all set;

python
import gevent

That’s it. You’re ready to play.

Lately I have been thinking of various ways to have a central configuration system for all our applications. A couple of the developers are working on a configuration service and are mainly looking at solving the requirements of the buy system first. They are going with the default key/value relationship and shouldn’t have much of an issue with that. This is an improvement to what the buy system had before, which did not scale well. But they are sticking with MySQL and I don’t think they are even thinking of other db options.

I like the idea of databases like MongoDB and how they store the data in a sort of document manner per key. I’m probably describing that badly. But the point being in old relationship terms, is that you can have any fields/columns for each record. So you don’t have migration issues when a new field is needing to be added.

The other way you can go and I think the developers are heading this way is to go with a straight key/value relationship, but that does not seem like it will be clean, especially when you go to multiple levels in the key/value relationships. Anyway this is a long way of me saying that I started looking at MongoDB, as I have heard a good amount about it and have always been wanting to try it out and see if it would work as I expected/hoped.

First impressions are good with it in that it was an absolute breeze to setup and get going with it. Granted just been using the javascript terminal to run the usual simple queries and get a feel for it. But man oh man,

• download
• untar
• create dir (/data/db)
• run mongod

and mongo db instance is running and waiting for connections. Granted this is locally and no replication etc etc happening as yet. Still need to work my way through all that to understand it better. But so far am liking it and thinking this may be a good start.

Next I want to setup up a simple web application and connect with it and maybe work on a simple configuration type app.

For the last couple of years I have been using Django, which is a very nice framework and it has all the bells and whistles. I’m very comfortable with it and can get things done quickly on it. But since I started playing with the AppEngine platform and developing a web application that used Django on AppEngine, I felt it was bloated and slow to start up. Ok granted Django, is not the best framework to use on AppEngine, although there is now django-nosql project in the works. So that got me looking at other Python web frameworks which has taken me from tipfy to webapp2. Both excellent frameworks that work well on AppEngine.

Now working on other platforms I am wondering about the other frameworks I have come across and trying them out. Flask being the latest. I like the documentation, which is extensive and am busy working through the nuances of the framework.

It is takes no effort to get coding simple applications. Busy looking at the effort to work on larger applications. There is documentation about this and for me the best way to work it out is to try it out. So once I have finished going through the documentation I’ll develop a web application using Flask.

The beauty of Flask for me, is that it is very lightweight but has the basic parts one needs. It is based on http://werkzeug.pocoo.org/ which is a WSGI utility library. It also makes use of Jinja2 by default for templating purposes, which suits me perfectly.

data = [
    [total_due, total_amount, '', 'BRAND', merchant],
    ['', '', '', 'DATE', datetime.date.today().strftime("%m/%d/%Y")],
    ['', '', '', 'ORDER NUMBER', order.order.order_number],
]
t = Table(data, style=[
    # left side
    ('GRID', (0, 0), (1, -1), 0.5, colors.black),
    ('BACKGROUND', (0, 0), (0, -1), HIGHLIGHT_BACKGROUND_COLOR),
    ('TEXTCOLOR', (0, 0), (0, -1), HIGHLIGHT_TEXT_COLOR),
    ('ALIGN', (0, 0), (-2, -1), 'RIGHT'),
    ('VALIGN', (0, 0), (-2, -2), 'MIDDLE'),
    ('SPAN', (0, 0), (0, -1)),
    ('SPAN', (1, 0), (1, -1)),
    # right side
    ('GRID', (-2, 0), (-1, -1), 0.5, colors.black),
    ('BACKGROUND', (-2, 0), (-2, -1), HIGHLIGHT_BACKGROUND_COLOR),
    ('TEXTCOLOR', (-2, 0), (-2, -1), HIGHLIGHT_TEXT_COLOR),
])

In the above the ’total_due’ value appears in the first cell of the table and I was wanting the first ‘TEXTCOLOR’ assignment to change the text color of that cell. Like it was for the second ‘TEXTCOLOR’ assignment.

The second assignment was working and setting the text color correctly, but not the first. WTF!

It finally dawned on me to see where/how ’total_due’ was being set, and that pretty much solved it. As it was being set with the following;

total_due = Paragraph("TOTAL DUE", styles['Right-Bold'])

So style[‘Right-Bold’] had a textColor attribute, so setting that to the color I wanted and we were good.

styles['Right-Bold'].textColor = colors.HexColor('#1b78d5')

The latest one I am playing around with at the moment is web.py. It appears simple enough, but crikey does it pack a lot. Webpy is capable of handling decent loads (used by reddit.com) and generally you would expect it to be bare bones. But not webpy you get form handling, it almost builds APIs for you out of the box and on top of it all it has a templating engine as well called Templetor.

Granted the templating is completely different to what I am used to. Mainly used to Django, Jinja2 which are both very similar to each other and Cheetah or ZPL which are XML based structure. You also have the option of using another templating engine and there are simple hooks in place already to make use of Jinja2 and others.

I’ve just gone through the documentation and it appears that it hits all the necessary points. Obviously using it in a complete web application would give a clearer idea.

The few things that I see lacking;

• Is a messaging wrapper, and this is just me being really lazy. It would be simple enough to write as a sessions storage mechanism is available.

• Is reverse lookup of routes. I am used to other frameworks that allow you to name the url path , and can then use the name instead of “hardcoding” the path in templates or reverse lookup the name to get the actual url path. Makes for easy development, but may not be good philosophically.

I also like that the documentation mainly goes about explaining how to solve issues.

https://github.com/reinbach/sd-postgresql

But when I got to the part of uploading the file and having to parse it, making use of xlrd to read the file etc, everything just seemed incredibly slow. So first thoughts were that I was doing something wrong, that my code was bad and needed to be optimized. Even though this was code that I had used in a very similar scenario to parse thousand line files and dump the data in a database.

This is not a very large dataset at all. So after digging around and looking for all the dos and dont’s, making use of batch querying etc. I still was not having much luck with speed. So this has me a little concerned about the app engine platform. Not wanting to spend too much time trying to optimize this like crazy at such an early point of the development cycle. I really just want to get this web app up and running as a prototype, I jumped back to using my good ol faithful postgres.

The main difference in the way/manner I was working with the file on the app engine platform was to pass the contents to the xlrd reader, instead of usually passing in the filename/path and letting it open the file. The file was only 3.7MB, so that may have been an issue. Also I could have made use of the BlogStore, maybe that would have been a better angle on handling the file upload/parsing.

I’m sure there is something I can do better on the app engine platform to make this a lot speedier, will just need to hunt down what that is. For now I’m plugging away and happy as a clam with postgres.

Jinja2 has solved some of these issues for me. Definitely have not had the chance to use it as much as I would like. Still on Django templates at CashStar, but everything else I am now using Jinja2. Slowly prepping CashStar to move to Jinja2. Probably the main issues in that it would probably be the most time consuming part of the transition is the use of ‘-’ within blocknames.

{% block page-js %}

works fine in Django, but is an issue in Jinja2, so have slowly been updating our templates to make use of ‘_’ rather.

A couple of the neat and helpful features in Jinja2 is being able to set a variable in the child template, which is then available in the parent template. The template you are extending. Actually there are quite a few neat things, besides the more powerful features of it. Best thing is to have a look at their documentation

webapp2 is super easy to setup, it is a single file and pretty much extends webapp. Include the extras, through in a couple of libs (eg Jinja2) and you are good to go. All the functionality you need to get going developing a half decent web application on app engine.

• add kids
• add allowances for the kids
• add transactions

The system will automatically track/calculate the balance. A cron runs daily and creates a transaction for those allowances that are due that day.

Still a little rough, need to add some pretty graphs and clean up the app in a general sense. But it is working and now I can track the kids allowances.

I made use of tipfy for the framework and did a bunch of unit testing on the models. Very happy with the framework and will continue using it for appengine applications.

Although lately I have run into some issues with trying to work with trying to run python as 32bit in one environment and 64bit in another. I must say it is all completely crazy trying to keep this all together, especially when you need to have other applications running at a specific architecture as well.

I had MySQL installed at 64bit with MySQL-python and all was working well with the virtualenv that had python at 64bit. But now I had to change python to 32bit, and that required me to “downgrade” MySQL to 32bit instead.

On a Mac (10.6) it is simple enough to switch between 32 and 64it and visa versa with the following commands

For 64bit
defaults write com.apple.versioner.python Prefer-32-Bit -bool no


For 32bit
defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

But sadly those do not affect the virtualenv, in order to modify the virtualenv python you need to do the following;

$ lipo -info /Users/cogg/.virtualenvs/tweakeats/bin/python
Architectures in the fat file: /Users/cogg/.virtualenvs/tweakeats/bin/python are: x86_64 i386 ppc7400
$ mv /Users/cogg/.virtualenvs/tweakeats/bin/python /Users/cogg/.virtualenvs/tweakeats/bin/python.old
$ lipo -remove x86_64 /Users/cogg/.virtualenvs/tweakeats/bin/python.old -output /Users/cogg/.virtualenvs/tweakeats/bin/python
$ python
[...]
>>> import sys
>>> sys.maxint
2147483647

That was found on stackoverflow

The base system handles has the following features;

• Ties up with the account/product levels created on CheddarGetter
• Account Management
• Handle Upgrading/Degrading account levels
• News
• Contact
• About
• Plan and Pricing
• Signup

There are a number of features and probably a few bugs that still need to be sorted out. It is a start and hopefully it will grow from here. Currently this base is being used on the http://www.noxls.com site

Well the whole exercise was very interesting to say the least. The whole Facebook development framework leaves a lot to be desired. There are a couple of issues that we just could not get around, mainly to do with SSL and Facebook really not handling that environment very well. The support is limited and misleading in places. No need to analyze this to death as it has been done plenty elsewhere by more informed people.

The cherry on the cake for me, was that the day we went live with our updated process, Facebook tanked and was out for a few hours. So that provided some exciting moments there. We now have a system wide kill switch for Facebook, which allows us to turn off any interaction we have with Facebook from the Connect functionality to the Like buttons.

Sadly they are the 800 lb gorilla and you have to work with what they give you. I ended up using a few of their APIs in implementing this all, which included the Javascript API, their new Graph API and the old Rest API. It would just be nice if they really tightened up their APIs.

The reviews have a rating, the ability to upload an image of the item/product being reviewed and are categorized. A complete admin section was created to help ease with the maintenance of the reviews and the site.

In developing the site I stuck with the original framework and believe I got a good understanding of it and how it works. I did miss some of the form functionality and the url management that comes with Django. Future AppEngine development will probably make use of the Django framework as a layer on the AppEngine framework, and I think I will get the best of both worlds.

You can’t beat the release process through the GoogleAppEngineLauncher, easy and straight forward. Although with the use of fabric on my other projects of late, I am not having any issue releasing to my own servers. Just a simple call. So I guess that is a moot point now.

So all in all good to get the initial project completed for a new framework/system and all the better that I still think the framework/system is solid and worth developing future projects on.

Then the projects that I was working on grew a little in size and the files were spread out in the file system and I started to find it a little tedious to move between these far ranging directories. At the same time I moved to a Mac laptop and was finding it a little frustrating having to use the Esc key to drive the Emacs commands, never mind my absolutely in ability to understand this lisp thing and setting up my .emacs file. So I started hunting else where for a new text editor to handle my needs and settled on TextMate.

Wow that was a really nice text editor and I was most happy. There was a couple of bugs/issues I ran into but was able to work around them. The main issue I had at first was editing remotely. I was using MacFuse at the time and there were a few instances where the file I was working on in the editor was touched/modified on the remote server and that caused all sorts of weirdness to the file in TextMate. I just ended up making sure to just code locally.

Although I did not always do my development on the Mac laptop. I still had my Linux desktop and so it was a bit of a pain moving back and forth between the systems and having to use a different text editor each time. I tried Geany for a while, as that gave me a file folder sidebar, and the plugins where helpful. But I never got hooked on Geany.

Well, lately the projects I have been working on lately have gotten huge in the number of files that they have, and TextMate was starting to have a tough time dealing with all the files. It is a known issue with textMate that it re-scans the files each time you return to TextMate. That got tiresome quickly enough. You can turn of the re-scanning, but then would I miss a file/dir change or what? I wasn’t bothered in trying to find out.

So enough was enough, and I decided to try Emacs out again, there is a Mac OS X version which I downloaded and that sorted out the Esc key only issue. And a little Googling later I had installed various modes etc and my .emacs file has grown ridiculously, but I am a happy clam again. I do not know what I was thinking or what I initially tried when I was first having issues with Emacs. It just seems this time, things clicked into place for me and I was able to find and setup decent modes.

Some things that I have learnt this time round, that are making my life real good, are;

• making use of frames. I now generally have a couple of frames open at a time
• yasnippet
• pareditb
• ido-mode. Absolutely fantastic in moving around dirs, opening files.

I do seem to have a better understanding of Emacs now, which is really helpful. Busy playing around with the org-mode. I have a number of feeds connected to various emac blogs.

Here’s to having settled on a text editor.

I really like the idea of the system being placed in Google’s Dataservers and leveraging their infrastructure. On top of that the hosting is free if your quotas are low and so only if your site becomes popular or is heavily used do you need to pony up with the cash.

To try out AppEngine I decided to write a simple Critic’s website and decided to use the “raw” AppEngine code/infrastructure. You can make use of 3rd party frameworks (eg: Django). The reason for that was just to get a good understanding of the basics and also to try something new.

Well it was simple enough and had some interesting way of doing things. For example when fetching a result set you always have to put in the number of entries you want back. I can see the reason for that, and so had to change the thinking/logic in the code in regards to that.

When creating/setting up an AppEngine application you need to create an app.yaml file that pretty much configures the site, and there are a number of options that are available to you. I really liked this as you are able to quickly and easily require that certain files/paths require authenticated users. You also map the paths to the necessary files in one location. This worked fine in a small application, so not sure how easy it would be to maintain it, if you had a very large application.

I sure have grown use to having all the bells and whistles that comes with using the Django framework, so will definitely be using that in the next applications/projects. It appears that the only part you really cannot make use of is the ORM that comes with Django, as you are using the proprietary storage mechanism that AppEngine is based on. Setting up the models was not that much different from Django.

The parts that I really missed in this application was the urls conf functionality. I like being able to change the url mapping in once place, which can be easily changed at anytime with out having to update the code everywhere. The other part was the handling of posted forms. I have grown lazy.

Well all that I have left to do on this project is to template it and it should be all good to go. A very basic site, but it allowed me to play with the AppEngine functionality and go through the complete cycle.

Well after reading a bit about Fabric, I decided to give it a try and am very ecstatic with it. All you need to do is create a simple fab file and call away. The Tutorial provided by them is straight forward.

I did end up modifying my sites in order to further simplify the releasing of code. I was using mod_python with Apache and had the media files located in the Web Document directory, but have now implemented WSGI and now the code released is all together and there is no need to keep the media files in a different location.

At the moment things are a lot simpler and it is now really easy to make a release. Further enhancements that I need to do is to add auto testing of the code base and ensure all tests pass, adding checks that code is all checked in. Maybe forcing a push to the main Repo etc.

I also want to add Nginx to the server setup as that is meant to be a much better setup. That will probably be the next step I take in regards to the server.

There are a dime a dozen of these To Do list managers, bug/issue trackers etc out there, but I have my own special way of doing things and I did not want to change my process. So I went ahead and created Track for myself. At the moment it is nothing fancy, it just does the usual.

Like creating/managing bugs per project. I can search on them, set milestones and assign the usual categories, priorities etc.

The neat thing is that it ties in with the Project section and indicates which projects are active. Well the active projects are those that have cases assigned to them that are active. Making use of the Track system will automatically keep the website up to date with the Projects I have worked on as well.

I have a netbook that was just setting there gathering dust and since Laura was complaining about not having a laptop, and always wanting to use mine. I thought I would give the distro a try. Previously I had been using Ubuntu Eeepc version, which was kinda sluggish.

Jolicloud’s website is very slick and helpful in getting the iso and with very clear instructions on installing it.

So after I figured out how to boot from the USB Stick I was up and running in no time. For some reason for my netbook to boot from the USB Stick, I need to add the USB Stick as a HD in the BIOS settings and obviously add it to the first media to boot from.

Jolicloud does require you to register an account with them, but this is probably due to the way they handle the managing the apps installed on your system. They do push the friends/social aspect as well, which is something I am not interested in. The other thing that I have not worked out is the lack of Flash in the browser. Looking in Synaptic it looks like the plugin is installed, but alas no luck with youtube.com etc. That is something that I need to get sorted out before Laura would even consider using the netbook as she likes her Scrabble on Facebook. The version I am using is labeled pre beta, so that could be an indicator that they are still getting things sorted out.

Other than the above issues this distro really does make the netbook feel a lot more responsive and usable.

I wrote a script that parses this Excel document and puts all the products into a database. The website allows a user to search the products and if they create an account to add products to a shopping cart.

The idea is that over time I will add more functionality, like order history etc.

http://assocbuyer.reinbach.com

I have a new job with CashStar, developing in python (Django) and loving it. Python is always a language I have admired and wanted to develop in. Over the years I have done some small projects and scripting, but now it is full time and I feel like a real programmer now. I can’t believe they actually hired me. Through all the technical reviews I thought I crashed big time, but alas I would get called for the next round of interviews.

So with what I am learning and my new found programming skills. I have gone and started rewriting everything in Django. Hence this site has gone and gotten a refresh, new code base and a new look.

I have been quiet of late as we spent sometime on the boat and then decided to stay where we are for a while longer, so I needed to get out there and find a decent paying job as well as getting settled into the new roles.

Not to mention that I have been crazy busy programming and getting things done, which is always great. More to come on these projects

I must admit this is not the prettiest as there is some duplication of information, buy as with most projects there was a time crunch and this is what I came up with in the allocated time.

First you have your usual menu code:

<ul>
	<li>
		<a href="">
			<span class="in">Home</span>
			<span class="out">Home</span>
		</a>
	</li>
	<li>
		<a href="">
			<span class="in">About</span>
			<span class="out">About</span>
		</a>
	</li>
	<li>
		<a href="">
			<span class="in">Contact</span>
			<span class="out">Contact</span>
		</a>
	</li>
</ul>

Here you notice that I have placed 2 span tags with in the link tags, this is where it is not pretty with the duplication of the link text.

To this we add the necessary styling:

ul {
	padding: 0;
	background: url('./images/background_menu.jpg') repeat-x scroll 0 0;
	height: 40px;
	position: relative;
}


ul li {
	float: left;
	list-style: none;
	padding: 0;
	margin: 0;
	line-height: 40px;
	position: relative;
}


ul li a {
	width: 100px;
	float: left;
	display: block;
	text-align: center;
}


ul li span.out {
	position: absolute;
	top: 0;
	left: 0;
	display: none;
	background: url('./images/background_menu_active.jpg') repeat-x scroll 0 0;
	width: 100px;
	float: left;
}

The position elements are what are critical in having this work correctly. So make sure you get those done properly.

Finally. to make it all work you have the javascript which makes use of jQuery:

$(document).ready(function()
{
	$('ul li a').hover(
		function()
		{
			$(this).find("span.in").each(function(){ $(this).fadeOut('slow')});
			$(this).find("span.out").each(function(){ $(this).fadeIn('slow')});
		},
		function()
		{
			$(this).find("span.in").each(function(){ $(this).fadeIn('slow')});
			$(this).find("span.out").each(function(){ $(this).fadeOut('slow')});
		}
	);
});

That’s it.

I was also not completely sold on it as a number of the templates and stylesheet code was/is stored in the database. This definitely makes it easier for the enduser to make modifications to these on the fly, but most endusers really are not interested in this and do not have the skills to, that’s why they have contracted the work out. The issue I find is that you do not have any version control of any changes to these files, you do have a log of which file was changed by who, but I do not see it providing a diff between these changes.

I believe that in upcoming releases they may be moving these files out of the database, so that would be an improvement to me.

The system definitely has a large number of modules that can be easily installed and there is a nice delivery system provided for those. Although it is hard at times to determine whether or not the module is solid and still being developed. As with most 3rd party development, the developer always starts out with good intentions, but may not complete the project.

All in all, once I got the understanding of the system the changes were relatively straight forward and mamaged to do what I needed to for the project.

In the past I have always made use of a full framework to develop websites, it didn’t matter which framework, whichever was being used or developing one from scratch. This was the first time that I approached a project and wondered whether it was necessary for all those layers for something so simple.

There are the obvious pros and cons for which ever way one decides to go.

A full framework allows the site to grow and expand with ease, you have all the layers in place, with a good separation between them. So all in all you have a tremendous amount of flexibility. The only con really is that you have a huge base for something so small, that it almost feels like bloat.

Without a full framework, the cons are much greater, like no separation between layers, the need to write those if you want them etc etc. Not able to expand very easily in the future.

Well in the ended I decided to go with the bare minimum as an exercise to see how things worked out. It was simple enough completing the project and getting it functional. It was nice having to go back and write code that you usually just take for granted.

The reason I went that way, was probably the need to break out of this mold of us webheads always wanting to put in the perfect system and will handle all future possible needs of the client, doing it the so called correct way, which we think is correct and not just getting the job done. The question is, how many times have we really had to go back and expand the system. Probably a lot of times, but have we not ended up re-writing things anyway? With the latest, coolest way?

That freaked me out, and had me hopping for a while, trying to work out what happened. It all boiled down to the data dir changing locations!

The quick fix was to just copy the old data dir into the new data dir location and restart postgres. All the databases reappeared, much to my relief. It is prudent to stop postgres completely before you move the data dir.

We made use of an already made module/component for Joomla and templated it. A few minor tweaks were made to the modules/components as well as a new module created to handle the stats.

You can get a glimpse of the site at zonelancer.reinbach.com

Over the next few weeks I plan to smooth out those edges and move it out of beta. Then over time add new features to it and grow the application. There is already a fair list of enhancements that I want to do, but first want to shore up the base and make sure it is solid first.

I would appreciate any help in testing the application, so if you have a moment or two, please head over to www.travelzlog.com and click around and then send me your feedback.

In the admin section the admin user has a configuration section where they can setup automatic levels for the postings to appear on the home page or for postings to be removed (deactivated) if a certain level of spam votes have been reached. The admin user also has the ability to manual manage the postings and setup the categories and subcategories.

I made use of Ajax to handle the the Category/Subcategory relationship in the form adding/editing a posting.

Each morning I do a world update on my system and I did not notice a few critical updates and I ended up with my apps not starting up cause they could not link to libstdc++.so.6. After a bit of searching I realized that I had actually upgraded gcc to 4.3.2 and this required that I do the system/world reinstall after setting up my current gcc version.

Well after the reinstall, which ran overnight, I re-synced and did my usual world upgrade/update and upgraded to xorg 1.5 which is a major change from the norm. Did I read the instructions completely, nope, I just went ahead and did the upgrade and low and behold my window manager was not fully operational to put it mildly. Anyway I went back to the instructions, which required that I update my INPUT_DEVICES and reinstall xorg server. I must say I was very thankful afterwards that I was required to run the eselect news command before the system would allow an update as that pointed me to the instructions.

Now me and my machine are happy campers.

The design was also a little narrow relative to other wordpress templates, but I thought it was a really cool looking design.

Although it is a little slower than my desktop, that is expected. It just required me to be a little more patient. The one issue that I did have, was not able to review my work via IE. I installed VirtualBox and moved across a snapshot of a Windows partition and was able to run Windows virtually on the netbook, but IE was not allowing me to view local websites running on the netbook. I probably do not have the correct IP or something along those lines, granted I made no effort to resolve this, but will in the next week or so, as that is something that I will need. Did not really need it while on the trip.

So all in all very happy with the netbook, can’t complain as it was very cheap and did what I wanted it to do. So now I am mobile in a big way.

I was planning on putting Gentoo onto it, but decided against that as it was suggested that compiling every package will be a bit resource intensive on the netbook so I decided on eeebuntu remix

The reason I decided to install a different Linux distr was to give me more flexibility and the desired environment for me. The default Xandros distro actually looked pretty decent and I could see anyone being able to use this with no problems. The interface was very good and intuitive. But I need to do some development on the machine and needed something more.

Well the only issue I had was booting from the USB Drive in that I kept getting a flashing cursor in the top left corner. The solution was that the USB stick’s boot partition needed to be fixed with the command

lilo -M /dev/sdd

After that it was pretty much plain sailing. The next part I’m working on is getting virtualbox setup, which is done, but really getting Windows installed within virtualbox and the trick is intalling Windows without a CD Drive.

Resources: Installing Ubuntu on USB Stick

Nice and straight forward to use, just a one liner to initialize the map and you have access to quite of bit of functionality through the API. I was able to make use of markers and polylines without any real issues.

At the moment I can connect an object on the page to a marker on the map, kinda, I make use of the latitude and longitude values. Just need to work out how to connect a marker to an object on the page.

Also just don’t forget to signup for your Google Map Key

Let me just say that I felt like an idiot at first. All the necessary libs were installed and being called correctly, but nothing was coming up. After much head scratching, I finally realized that I needed to provide styling for the slider for it to show up.

So at the very least have something along these lines;

The idea of the javascript snippet of code is that it can be placed in a form that has a select list pulling data from a lookup table and would allow the user to add to the lookup table without having to leave the form.

The javascript code sends a request to a PHP file on the server that adds the data to the database. A new list of the categories is returned in JSON format and that is parsed into the new options list for the categories.

Not only does this help mimimize the mistakes I make, but it definitely makes life a lot easier just running the script whenever a change is made to a table that needs auditing. So that is a win win.

The idea would be to create a number of other scripts that allows the database to be re-created competely, this is for development purposes and allows for changes to the database to happen rapidly and with minimal mistakes. But that is for another posting.

When adding the article it is parsed and html syntax is added to the text based on the markdown syntax. This method then only requires the text to be parsed when it is added/updated and not everytime the article is viewed.

I wanted something that would be able to display code snippets more intuitively and effectively. The article app includes an extra field for the html content that takes the text/body of the article and converts it into an html structure.

The conversion process makes use of markdown and codehilite and the processing is done on the server side of things and only has to be done when I add/update the article, instead of at the client side each and everytime the article is viewed.

I have written a short article on it that can be found here

The auditing steps are implemented in the database layer through functions and triggers making use of PL/SQL, it includes a login function that creates a session table that is used to ensure that the user is logged in and that the auditing function has access to the users information.

But in wanting to make use of JQuery more and to get more up to speed with it, I came across Flot and decided to give it a whirl on the Powerball app.

For more information head on over to the Flot article.

The issue I just ran into was with me making a tweak on a django installation and modified the urls.py file, but in doing so I mispelt a module it needed to import, and alas the site is currently failing to import this module now and errors out. Well the problem is correcting that mistake is easy enough, but the web server is not re-reading that file until after a certain number of requests on the site.

Normally if you have control of the web server you can reload/restart it, but here I have to let it run its course and with time and enough requests it will correct itself.

Anyway it is back to normal now, so no need to worry anymore. Just have to be more careful in future.

Normally this would not be an issue as the dev environment would match the production environment and these issues would be sorted out in the dev region where you have absolute control and then the release process would minimize the human element as much as possible and this sort of error would not occur.

At the moment I make use of mod_python in the dev environment and have all requests re-reading the files. While on the pyton.reinbach.com site I am making use of Fast CGI and a slightly different setup.

So for a mini project I played with the powerball lottery numbers and created a very simple application that pulls the numbers from the powerball lottery website and dump them in a database and then extract some data from the database and present them.

All that has resulted in a couple of pages that can be found at python.reinbach.com/powerball. These pages show the results and explain with code snippets the underlying code.

I made use of django for the web framework and MySQL for the database and python for the script that fetches the numbers from the official powerball website.

The projects were nothing too involved, just reading Excel files, parsing the data, saving specific records, saving the rejected records to a file and emailing the results to relevant parties.

I made use of a lib called xlrd [http://www.lexicon.net/sjmachin/xlrd.htm], that provides functionality to read Excel files. This definitely opens up a large amount of opportunity with Excel files now, seeing that I never found anything similar for PHP on a Linux platform. No longer have to ask that the files put converted to a text format.

Hopefully I am able to incorporate Python in future projects as and where appropriate, it is a nice language that I really enjoy.

There were a few things that needed to be named correctly to make the most of the framework, which was interesting, but no big deal. As they do a good job of providing some decent documentation on the system and had a decent tutorial.

The version I used was the current stable release [Stable: 1.1.19.6305], but it appears that the latest release is fairly standard with some new features/improvements. The main thing that I noticed in the time I worked with it between the versions was the Form helpers being separated from HTML helper.

The project was pretty straight forward in just being a dynamic website with the usual CMS functionality.

I definitely want to try the framework out on a more demanding scenario and hopefully that will be happening soon.

So it is a bit rough around the edges at the moment, but it is something I have put together and will fine tune overtime.

It actually was not hard at all, just had to do a little googling and everything panned out for me. So below are the steps I took.

Step 1: remove dependency on RHN for updates/upgrades etc. To make use of these repos you need a subscription and I am just not into that. rpm -e yum-rhn-plugin rpm -e redhat-release-notes-5Server redhat-release-5Server –nodeps

Step 2: get the centos release and notes rpms that get things going for you in moving from RHEL to CentOS. You need to know what platform you are running on 32bit or 64bit

A quick command to find that information out is; uname -a

i386 (32bit) wget http://mirror.centos.org/centos-5/5/os/i386/CentOS/centos-release-5-1.0.el5.centos.1.i386.rpm wget http://mirror.centos.org/centos-5/5/os/i386/CentOS/centos-release-notes-5.1.0-2.i386.rpm

x86_64 (64bit) wget http://mirror.centos.org/centos-5/5/os/x86_64/CentOS/centos-release-5-1.0.el5.centos.1.x86_64.rpm wget http://mirror.centos.org/centos-5/5/os/x86_64/CentOS/centos-release-notes-5.1.0-2.x86_64.rpm

Step 3: run the rpms, this will setup your system to make use of CentOS and actually updates you repos list (/etc/yum.repos.d/) rpm -Uvh centos-release*.rpm –test rpm -Uvh centos-release*.rpm

run the test first to make sure you do not have any dependencies, if you did you will need to get hold those files to fulfill the dependencies. I did not have any issues so I just continued. Good luck if you did have dependency issues : )

Step 4: (optional) update the name of the system. The previous command will result in this file being updated, it really is up to you. I went and changed mine back to ‘RedHat Enterprises Linux Server release 5 (Tikanga)’ nano /etc/issue

Step 5: lastly I went ahead and did an upgrade: yum upgrade

That resulted in 334 packages needed to be downloaded and installed. After that was complete, everything worked fine for me and I was able to install the necessary files that I wanted without any issues. Thank goodness. Now I do not have to worry about mounting the CDRom and I have no problems using yum accessing open repos.

Resources:

• http://jyrxs.blogspot.com/2008/02/using-centos-5-repos-in-rhel5-server.html

Once up and running you have almost everything you need. You get all your standard applications to do almost everything you need. Now to do development you need to take an extra step and update to the ‘devel’ stages/packages. A nice feature is that the updates are seamless and happen in the background, after a simple click.

The one thing that is missing for me is the lack of GnuCash, I need that program, well I just really want it. I have a long history with it. Yes I could go ahead and download the packages and install it myself, but the better thing would be to follow up on the bug tracker and help to get it into the list of applications, there appears to be a couple of bugs still open for it at the moment (FL-735 and FL-1251).

A little weird issue I do have is that my /etcX11/xorg.conf file does not appear to be read on the initial load, I have to login and out first before my dual monitors are correctly picked up.

So today I was browsing DistroWatch again and getting the urge to try another distro and the 2 that caught my eye are Mint and Sabayona

I have an AMD64 system so that is something that the distro needs to be strong in, which looks like Sabayona is, seeing that it is based on Gentoo which I have used and really enjoyed, that has been my favorite distro for a while now. I really like the way you get down and into the meat of it when working with Gentoo. But I am going to wait till Sabayona has made it’s latest release, which seems that they are close on doing that. It is meant to be at the end of this month (June 2008) So I’ll go ahead and try Mint next I think.

When I have finished a project or have time during a project, I try to remember to move any improvements back into the base, I really ought to do this more often, but alas. The improvements are limited to the functionality that the base has, I do not want to have a bloated base to start with, I much prefer a very simple clean code base to start with and prefer to add rather than take away.

Currently the base has the following;

• MVC Framework
• DB Abstracted Layer
• User management
• Content
• Test Scripts

I am quite tempted to remove the Content piece, but so far it has managed to survive where it is at the moment.

Currently the DB Abstracted layer is MySQL specific, but I have been working on a side project that makes use of PostgreSQL heavily and I have extended this layer to handle both DB Sources. Something I need to move back into the base.

The Test Scripts are individual unit tests run through PHPUnit and test individual functionality, I need to get together a Test Suite that runs the whole bang shoot together.

So at the moment, this is just a blog entry to get things moving along. But plan to create a space here to better share my code and scripts.

Finally the code base can be found at http://www.reinbach.com/code.tgz any feedback would be interesting.

PL/PGSQL is the language I used to program these functions and triggers. I was looking at PL/Python but decided on PL/PGSQL as it is more ’embedded’ in Postgres and you have access to more information, while with another Language like PL/Python I would need to make more calls.

One gotchas is that one needs to look out for is the NULL value. It can really wreck havoc on your functions/triggers. If you have a NULL value for any var in your statement you’re ending up with a NULL result!!

To really make the auditing work well you need to know who is taking the action and for that I required a change in the normal login query, now instead of just checking that the user exists in the database I now create a temporary ‘session’ table that holds the username of the user and so when an auditing action is invoked this ‘session’ table is queried for the username, if it does not exist the complete query will fail. Which could be considered another layer of security at the database level, albeit a very very weak layer that is easily circumvented by just manually creating the ‘session’ table.

The auditing record is tied back to the original record by the table’s primary key and this is where one is required to make a slight modification in the table structures so that there is always only one primary key. This just makes things simpler for the auditing record and connecting it back to the record. I have not run into any issues with this at the moment. For those tables that had primary keys made up of more than 1 field, I changed by adding a system primary key to the table and making a constraint requiring the previous primary keys to be unique together.

So the auditing table is made up as follows;

• trx_type: the type of transaction that occurred (INSERT/UPDATE/DELETE)
• trx_table: the table that the transaction occurred in
• trx_key: the primary key of the table/record
• trx_time: time the transaction occurred
• trx_user: the user responsible for the transaction
• trx_desc: an xml string of the values affected

To provide a clear indication of what happened I make up a XML string showing the new/old values for each of the fields being affected. This allows easy parsing in the future to provide a clear history of the record.

In order to make life easier for me I have created a simple script that generates the function/trigger. I just pass in the table name and the primary key and it queries the database for the table information and then creates a file that holds the function/trigger. The reason for putting it in the file is that I just need to run the script once to generate the file and then I have another script that regenerates the database for me, making use of all these files that hold the functions/triggers plus a few other files that hold the data structure and default data (I’ll post this in another posting). This allows for rapid rebuilding of the database as well as being able to create multiple instances of the database which I do for unit testing purposes. Just make sure not to add a trigger to any type of ids table that is used by the auditing table, otherwise you’re in a nasty loop. Yes I did that, and it took me a little while to work out what the heck was going on.

Well that is a simple overview of the auditing that I have implemented. Granted it is still early days for this and it will more than likely go through some growing pains.

What made me get this done with little procrastination, was that the application fee is going up by about $300 after 31 July 2007. So as you can see, I just managed to squeeze in before that date. Not sure exactly how long the whole process will take, but it should be about 6 months or longer.

I’m having to apply for a new passport, seeing that mine is expired. So I am going to be running a little experiment between the services of the 2 governments. I have found it a real pain to deal with the SA Embassy about passports. I have applied in the past for a new passport and a temporary passport, but only received the temporary passport and never got my permanent passport. I have tried emailing them, but the posted email address bounces back.

So I am going through the process again. I started about a couple of weeks ago, sending off for BI-9 forms, which are for finger prints. Those have just arrived, I’m going to have to get my finger prints taken and fill in a couple of forms and will be sending that off. Must mention that I sent a self-addressed stamped envelope for them, but it appears that the envelope I sent was too small, as they sent a large manila envelope with the BI-9n forms in it, so that was nice that they did that.

So we shall see which we get completed first… US Citizenship or SA Passport… let the games begin.

But am very glad to have gotten the majority of the things to the new place. We are spending our first night at the new place, bed is setup, and am loving it. But I’m tired now and am ready to crash.

Tomorrow will just be returning the Rental truck, hit Costco for a couple of things and maybe, if I’m up to it, making that last run to the old place for the last of our things.

Home Sweet Home!

It’s a little further from work, so will not be able to pop home for lunch, which is sad. But the place is much larger and we have a decent size yard for the kids to run around in. There is also a pond that has fish, turtles and ducks, so that will be fun as well.Sadly no pool, but we hope to install one in the following year or so. Probably going to get one of those above ground pools which should be funny, but some relieve from the summer heat.

We’ll be moving a few things during the week and then get the big things moved this weekend. The ability to take a bunch of trips is great, we’re use to having to pack everything at once.

Over all the management part is better, I think, it is easier to move around and get things done. I still need to sort out the gallery as that is only partly installed at the moment, I have run out of time and will get that done when I get a moment to.

This is the first blog entry in the blog system that comes with version 5.1, not sure if there was a blog system in the previous version.

I have added the following cron to my crontab to perform a sync on a regular basis so that I keep up to date.

sync gentoo updates

min[0-59] hour[0-23] day-of-month[1-31] month[1-12] day-of-week[0-6]

15 3 * * 1,3,5 root emerge –sync

This cron will run Monday, Wednesday and Friday at 3:15am. This saves me having to perform a sync manually.

Then I can perform an update when ever I want to knowing that the portage list is pretty much up to date, with out having to wait for a sync to happen.

emerge world -uva

The ‘-va’ options displays in verbose fashion all the USE flags that will be used for each package as well as prompting you to perform this emerge or not.

Then periodically I run the following commands to keep the system clean.

emerge –update –deep –newuse world -va

emerge –depclean

revdep-rebuild

See the Portage Introduction doc for complete information on portage.

Or for all Gentoo Docs

The rest of this page is the steps I take to backup a DVD. At the moment there are a couple of types of DVDs out there which range in size the normal DVD is about 4.7Gig and I believe this is a single layer DVD and then there is the double layer DVD which provides you with about 8Gigs of space. So if you are backing up a DVD make sure you have DVDs that can hold the DVD you are going to backup.

Step 1 - Copy DVD $ vobcopy -m

This copies the DVD onto your sytstem and creates a dir with the name of the DVD all in ippercase. So make sure you run this command from the dir you want the backup dir to be placed in.

Step 2 - Create Image $ mkisofs -dvd-video -o dvd.img DVD_DIR/

DVD_DIR/ is the directory that was created by the previous step. It’s good to perform this step as it helps shrink the file size to be able to fit on the DVD and makes for a faster burn.

Step 3 - Burn It $ growisofs -dvd-compat -Z /dev/dvdrw=dvd.img

And that should be it, with you now having a nice backup of your DVD

Good investment properties are identified as those that are able to be rented out at a certain rate of return over and above the mortage of the propery in relation to the amount of money invested.

For example;

• Down Payment $20,000
• Property Cost $100,000
• Mortgage $600
• Rent $800

Rate of return is calculated as follows; (Rent - Mortgage) / Down Payment (800 - 600) / 20000 = 0.01 Annualized it would be 12%

Will be using the following source for mortgage calculations as they are very well layout out and explained. http://www.hughchou.org/calc/formula.html

• mortgage calculations
• future values

The tool will need to be given a number of critieria in order to be able to run its searches. These criteria are identified as follows;

• region (zip code(s))
• desired percent return
• down payment
• percent of property cost down payment to equal

A couple of possible alerts;

• when the property is no longer available
• change in price of property

Best option would be MLS, but need to get access to all the various MLS listings around the country, could be expensive and lots of programming for each one

Another option is to just scrap off realtor. Not all properties are listed here though, but may be able to be used to identify good regions

• http://forums.gentoo.org/viewtopic-t-117709.html

Note: FILENAME - is the name of the dvd you are creating

Step 1 - Transcode

Transocde AVI file into dvd friendly format. $ transcode -i FILENAME.avi -y ffmpeg –export_prof dvd-ntsc –export_asr 3 -o FILENAME -D0 -b224 -N 0x2000 -s2 -m FILENAME.ac3 -J modfps=clonetype=3 –export_fps 29.97

The above script is for a 16:9 aspect ratio, if you need that to be 4:3 aspect ratio then change the following –export_asr 3 => –export_asr 2

This results in 2 files, once for the video and the other for the sound

Step 2 - Combine

Need to combine the video and sound file together $ mplex -f 8 -o FILENAME_dvd.mpg FILENAME.m2v FILENAME.ac3

Step 3 - Menu

Create an xml file (this is one without a menu) <dvdauthor dest=‘DVD’>     <vmgm />     <titleset>        <titles>           <video widescreen=‘nopanscan’ />           <pgc>              <vob file=‘FILENAME_dvd.mpg’ chapters=‘0,0:30,1:00,1:30,2:30,3:00,3:30,4:00’/>           </pgc>        </titles>     </titleset> </dvdauthor>

Step 4 - Author

This run the xml file through dvdauthor $ dvdauthor -x dvdauthor.xml

Step 5 - Burn

Create DVD image that will be used to burn DVD $ growisofs -Z /dev/dvd -dvd-video DVD/

That is is. Except that I should create some scripts for all this and maybe package it into a simple program

I used the following resources for help;

• http://gentoo-wiki.com/HOWTO_Dual_Monitors

So after much tweaking and playing around below is the part of the xorg.conf file that setup the monitors the way I wanted them.

Laptop — snip —

Section ‘Device’     Identifier   ‘video0’     Driver   bsp;  BusID   bsp;  Screen   bsp;  Option   itorLayout’    ‘CRT,LFP’    Option   e’   bsp;  Option   Info’   bsp;  Option   ce’   ‘false’    Option   bsp;  bsp;  Option   bsp;  dSection

Section ‘Monitor’    Identifier   ‘Laptop Monitor’    Modeline   ‘1280x800@60’ 83.91 1280 1312 1624 1656 800 816 824 841    Option   dSection

Section ‘Screen’    Identifier   ‘screen0’    Device   bsp;  Monitor   Monitor’    DefaultDepth   24    SubSection    ‘Display’       bsp;  bsp;     bsp;  ‘1024x800’ ‘800x600’ ‘640x480’    EndSubSection EndSection

— snip —

The default xorg.conf file had ‘vesa’ as the driver and I changed that to i810’ (I just had to change the text here to ‘i810’, and it worked for me) ‘Clone’ is set to false as I did not want a duplicate copy of the desktop on my external monitor, but rather to extend it.

External Monitor — snip —

Section ‘Device’    Identifier   ‘video1’    Driver   bsp;  BusID   bsp;  Screen   bsp;  Option   itorLayout’    ‘CRT,LFP’

Option ‘MetaModes’ ‘1600x1200’

EndSection

Section ‘Monitor’    Identifier   ‘NEC Monitor’    HorizSync   31-81    VertRefresh   56-75    Option   dSection

Section ‘Screen’    Identifier   ‘screen1’    Device   bsp;  Monitor   EC Monitor’    DefaultDepth   24    SubSection    ‘Display’       bsp;  bsp;     bsp;  bsp;      Viewport    0 0    EndSubSection EndSection

— snip —

Here I added the external monitor’s settings. You ned to know/determine the HorizSync and VertRefresh that is specific to your monitor, a quick way is to boot up a Gnoppix CD and see what it sets for your monitor or look at the specs for the external monitor.

For the external monitor I needed to make sure that the ‘DefaultDepth’ value had a matching ‘SubSection’ with a ‘Depth’ that matched it.

Once I had everything setup I ran into the issue that the external monitor’s desktop would scroll, so that it appeared that I had this really large desktop but only a portion of it would show on the monitor and moving the mouse to the edge of the monitor would result in the desktop scrolling in that direction. I did not like that and so that is what the ‘Viewport 0 0’ line does it. it stops the scrolling.

Monitor Layout — snip —

Section ‘ServerFlags’    Option   ‘Xinerama’   ‘on’    Option   ‘RandR’   ‘on’ EndSection

Section ‘ServerLayout’    Identifier   ‘Default Layout’    Screen   bsp;  ‘screen0’ 0 0    Screen   bsp;  ‘screen1’ RightOf ‘screen0’    InputDevice   ‘Generic Keyboard’    InputDevice   ‘Configured Mouse’    InputDevice   ‘Synaptics Touchpad’    Option   erama’‘on’ EndSection

— snip —

The last section pretty much determines how the monitors play together. So that is it, with these settings for the monitors in your xorg.conf file you should be all set with dual monitors.

The latest news is that I am now working for INTECH as a Web Application Developer. I must say I am extremely happy at the moment. The people there are extremely knowledgeable in their fields and the environment is great.

On the personal side Laura and I are going to be attempting to do some Triahtlons etc, this is really a goal we have set, that will see us getting back into shape. It has been a while since we were last in shape. I have been running a bit lately, so we are making slow progress, but progress nevertheless.

We have just bought a minivan on eBay of all places and need to pick it up next week, so that is exciting. Sadly that will see us retiring the white subaru, which has served us well over the last few years. Sadly the lack of airconditioning is a big issue here in Florida, go figure. Also the body is taking great strain with rust appearing everywhere. And not to mention the hubcabs going walk about. Lost one, but was able to recover the other as I saw it rolling down the road!

That’s it for news, have some other content to get up.

Almost got the UHaul stuck in the driveway, but after a few planks and bakc and forths managed to get the truck back to the beginning of the driveway and ended up having to carry everything through the mud to the UHaul and packed it.

Spent most of the evening getting the heavy things packed with the help of my colleagues. Then spent the next day finishing up the “small” stuff and cleaning, which ended up taking the better part of the day. We packed everything in the UHaul and then thought about staying the evening and leaving in the morning, but I refused to unpack anything, so we finished cleaning up and started our trip at 6pm!! Drove to North Conway and had MacDonalds for dinner!!! Then really started on the trip.

We made it to Boston that evening and stayed there, so a whopping 2hrs first leg.

Well the next day we pretty much double that to 4hrs and have made it to NY and are staying the evening with Linda and Scott. Tomorrow I think we will do a little further, and before we know it we will have made it the whole way. No need making it torturious(sp?).

So at the moment we’re at the Warnash’s and having a great relaxing time with them.

I then decided to see what it would take to port JaG CMS from MySQL to Postgres, and was really impressed with how little it took. I did actually find a couple of little things as well as bug or two in my sql queries. But it really did not take much time at all, less than an hour I would say. Most of it trouble shooting my bugs and working out that Postgres does not like integer statements in the where clause (eg: WHERE 1) but rather that it needs to be a boolean statement (eg: WHERE 1 = 1)

So all in all I was impressed with Postgres and hope that I willmake more use of it in the future.

I realize that MySQL is/has developed a lot of the functionality that Postgres already has, like Stored Procedures, Triggers etc. but I don’t think it hurts to know a number of databases really well.

We have developed an event management system for AA Events, who have 2 events each year one Ski event in Vail, CO and a Golf event in Newport Beach, CA which has now moved to San Diego, CA.

The system handles registration of attendees to get the people into the system then the staff have a number of sub systems in the backend to fulfill their tasks which range from assigning Flights, Hotel accommodation, ballroom seating through, silent and live Auction system to a section for Sponsors to enter in their information and seat their people. There is actually a multitude of functionality in the background which has grown and been tweaked over the course of the last few years.

The registration part is being refactored as that is not working how we want it to, there are still a couple of user issues we are running into to. At the same time the front end of the website is going to be redone, so that it ties in more closely with the backend and there will be a smoother transistion from there to the registration for the various events.

Did manage to have one day of relaxation with the family in FL, would have liked that to have been more but not the case this time.

So you see it was a business trip to Vail, CO. The skiing was good, but sadly there had not been any snow for the last week or so when we arrived, so it was well groomed runs for the most part.

Now which is better and which to spend one’s time learning? Here I have been looking for some decent Python Web framework and suddenly come across 2. Well it does appear the DJango attempts to handle all aspects of the framework, while TruboGears brought together a number of pieces to create the whole.

So each way has it’s pros and cons, both of near version 1 release. I’m going to spend sometime with both, which will help me to learn both of them as well as at the same time getting more comfortable with Python and how it does its thing.

I have TurboGears setup already and have started playing with it, so I’ll run through a few more tutorials on that and give it a go, after that I’ll move onto DJango and see how that feels.

Well that was until I came across TurboGears [http://www.turbogears.org] Very much along the lines of Rails. Just set it up on my machine and going through it, and so far it is very impressive. Maybe this will be the tipping point for me to really get to learn Python.

Been working on JaG CMS of late and have a nice code base there from which to start most projects. So spent the last week working out the best way to extend this code base, and ideally that would be through modules.

I had a couple of requirements for the modules that I wanted to achieve, which were;

1. minimal impact on the code base. meaning I did not want to have to update the code base whenever a module was installed. The code base needed to exist as it was. I didn’t mind making changes to the code base to handle modules, but that was to be a change for modules as a whole to make use of the code base.

2. 1 location of module files. This would lead to easy management and maintenance of each module

So spent the week creating a catalog module and have most of that achieved with the requirements above. Just need to complete the coding of the module, which should not take much longer, but the module code is sitting in a separate directory from the code base, and is interacting with the code base as needed.

Will be spending this week completing it and then creating some sort of managment interface to modules. After that I will be working on plugins, how to create those and have them interact with the code base.

Have moved onto the development of modules for JaG CMS, wanting to make it simple and easy to develop/create modules that can be added and removed easily from the core code base.

The first module that is being worked on is a catalog module with the idea that this will a part of an online shopping system that sits on JaG CMS.

Now to let developers know about it so more testing can be performed and some feedback generated. That should help us to make sure it is a decent product.

Been interesting in working out how to handle the transistion from no libraries available at the beginning of the installation to a gradual addition of libraries of the code base as parts are set up.

For example have no templating engine available to handle the presentation logic, but after the first step this is available. The step after that will add the db abstraction layer, as the parameters of the db are provided.

So have found that interest in solving. But now that I have the design it is flowing easily together.

It is a simple website CMS system, that allows a user to manage a website, by adding/editing pages and content. Have not made a code release as yet, but that should be in the next week or so. Having to complete the installation script and we should be able to make the first release.

For more informaiton you can find it on http://sourceforge.net/projects/jagcms

On a new schedule that sees me getting up way before the crack of dawn. Now getting up around 4am, I like this time of day as it is very quiet, most of the family not running around like crazy, so I can get some decent work done. This also sees me crashing real early in the evening.

Have a number of ideas and am looking to get them implemented this year.

What is news. Well this site is damn old and need of a major update. So first things first that will be the album, as the wife is asking for that, and so that makes it the highest priority. Yes I know, I know.

Will be back shortly with the album. Till next time.

Well well what can I say. We have since moved to Fryeburg, Maine and winter is fast approaching.

Family is all well and good. With the new year rapidly approaching I think I will get a jump start on resolutions and start with me keeping this site more up to date.

• Idaho, really liked Sandpoint
• Washington
• Oregon
• California, drove the whole coastline, blew through San Francisco and popped inland to Lake Tahoe and Yosemite, back to the coast near Monteray Bay down along the rest of the coast line, spent a week in San Diego and visited Sea World, San Diego Zoo and the Wild Animal Park
• Nevada
• Arizona
• Utah I think Ethan watched Dumbo a 100 times and never saw any of the scenery.