Challenge: Exporter for Prometheus
==================================
This is an prometheus metrics exporter for the soundcloud challenge. 

Requirements
------------

* Docker, of course ;-)
* Working go 1.8 (older should work as well)

Build
--------------------
Build binary `exporter`:
```bash
make
```

Build docker image with tag `challenge/exporter`:
```bash
make docker
```

Run
---
Start locally in docker: 

```bash
docker run -d -p 8080:8080 --name challenge beorn7/syseng-challenge
docker run -d -p 8081:8080 --link challenge --name challenge_exporter challenge/exporter --endpoint http://challenge:8080/stats

docker stop challenge_exporter
docker rm challenge_exporter
docker stop challenge
docker rm challenge
```
or with docker-compose
```bash
docker-compose up
```

Metrics should now be available on `http://localhost:8081/metrics`

Bonus
=====

1. What are good ways of deploying hundreds of instances of our simulated
   service? How would you deploy your exporter? And how would you configure
   Prometheus to monitor them all?
   
   Pretty easy with *kubernetes*. 
   Just run the exporter along the app in a pod with an ReplicationController:
   
   _Note: Config is just an proof of concept, not fully tested:_
   ```yaml
    apiVersion: v1
    kind: ReplicationController
    metadata:
      name: replicatedapp
    spec:
      replicas: 100
      selector:
        app: exportedapp
      template:
        metadata:
          name: podapp
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "8081"
          labels:
            app: exportedapp
        spec:
          containers:
          - name: challenge
            image: beorn7/syseng-challenge
            ports:
            - containerPort: 8080
          - name: exporter
            image: exporter
            ports:
            - containerPort: 8081      
   ```
   
   Just use the service discovery in prometheus:
   ```yaml
   - job_name: kube-app  
     kubernetes_sd_config:
       - role: pod
     relabel_configs:
       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
         action: keep
         regex: true
       - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
         action: replace
         regex: ([^:]+)(?::\d+)?;(\d+)
         replacement: $1:$2
         target_label: __address__
       - source_labels: [__meta_kubernetes_pod_name]
         action: replace
         target_label: pod       
   ```
   
   DNS discovery may be an alternative, for example with coredns.
   
2. What graphs about the service would you plot in a dashboard builder like Grafana?
   
   Usually graph everything where attention is required.
   It does not make sense to monitor metrics/graphs where nobody needs to get in action. Less is more.
   
   Assuming we have a fleet of this service and monitor all of them, it makes sense to graph in groups.
   
   *Graph* Request rates per code (QPS):
    ```
    sum(app_request_rates) by (code)
    ```

   *Graph* Highest latencies:
    ```
    max(app_duration_avg)
    ```
    
    *Singlestat* Running instances:
    ```
    count_scalar(app_up == 1)
    ```
   
3. What would you alert on? What would be the urgency of the various alerts?

   High: Too few apps are up (to handle all requests)
   Middle/Hight: Request times are too high (priority depends on latency)
   Middle: Too many bad/failed requests (5xx) codes in comparision to suceeded (2xx)
    
4. If you were in control of the microservice, which exported metrics would you
   add or modify next?
   
   Depends a little bit on the service, but probably these will be useful:
   - CPU/RAM utilization. Probably network throughput. 
   - Avg duration time per code, method.
   - Request rates per code and method.
   
   In general, monitor more metrics than you need in the moment. 
   As more than you have, debugging an issue can probably solved by an metric which is not active monitored.